While defining a struct for reading JSON text, we normally define a one to one relationship between the JSON field and the struct variable.

I start by converting the JSON to a Golang struct. My favorite JSON to Go converter is: https://mholt.github.io/json-to-go/

The JSON can be unmarshalled into that struct directly using the encoding/json package in the stdlib. Note that the struct has to have exported fields defined within it i.e. capitalized field names.

https://golang.org/pkg/encoding/json/

Here is an example to unmarshal JSON data that is well defined.

type Response struct {     
   Page   int      `json:"page"`     
   Entries []string `json:"entries"` 
}

str := `{"page": 1, "entries": ["classA", "classB"]}` 
   
res := response{}     
json.Unmarshal([]byte(str), &res)

See it in action: https://play.golang.org/p/7IEki4G7AEA

However, not all JSON data is well defined.

What do we do if we have to unmarshal JSON data that does not completely adhere to the struct we define?

We could custom unmarshal this JSON data into a struct that we define by writing our own custom unmarshal function.

E.g. The crypto currency exchange GDAX market feed API.

This API has an endpoint get-product-order-book that returns JSON data that gives us bids and asks for a crypto currency. These values are returned as array within an array with three floats and the fields are not well defined.

https://docs.gdax.com/?javascript#get-product-order-book

Values returned are:

{ 
"sequence": 3977318850, 
"bids": [ 
          [ "4625.78", "0.80766325", 3 ] 
        ], 
"asks": [ 
          [ "4625.79", "3.0154341", 3 ] 
        ]
}

The data within the array actually represents the following three values:

[price, size, order_id]

The JSON converter gave me this golang struct:

type AutoGenerated struct { 
 Sequence int64 `json:"sequence"`  
 Bids [][]interface{} `json:"bids"`   
 Asks [][]interface{} `json:"asks"` 
}

Ideally, I would like to define the struct as below and Unmarshal the API contents directly into this struct.

type Bid struct {     
 Price     string     
 Size      string     
 NumOrders int 
} 
type OrderBook struct  {    
 Sequence int64 `json:"sequence"`   
 Bids     []Bid `json:"bids"`    
 Asks     []Bid `json:"asks"` 
}

But the Unmarshaller will not automatically unmarshal into the above defined struct.

E.g. The Unmarshal code below using the struct, throws this error : json: cannot unmarshal array into Go value of Bid

body, err := ioutil.ReadAll(resp.Body)  
 if err != nil {    
 fmt.Println(err) 
} 
var orders OrderBook 
err = json.Unmarshal(body, &orders) 
if err != nil {   
  fmt.Println(err) 
}  

Unmarshal calls the UnmarshalJSON method of the value. Hence, for the above conversion, we can define a custom UnmarshalJSON function  for our array to struct conversion. Since we want the inner struct Bid to be unmarshalled correctly the method has to be defined on that struct.

func (b *Bid) UnmarshalJSON(data []byte) error {     

 var v []interface{}     
 if err := json.Unmarshal(data, &v); err != nil {       
    return err     
 }     
 
 b.Price, _ = v[0].(string)     
 b.Size, _ = v[1].(string)     
 b.NumOrders = int(v[2].(float64))    
 
 return nil 
}

We Unmarshal the data into an interface and then we can set the values as desired with a type assertion.

This will cause the  Unmarshal on the main struct to work seamlessly!

The full code is available on github at https://github.com/mariadesouza/go-orderbook.

Partial Unmarshal of JSON

At times you want to read a JSON file and extract just a few fields.

If you want to avoid defining a struct, you could decode it into a map of generic types and cast them appropriately.

b := []byte(`{"earnings":1000.50,"names":["Joe","Jane"]}`)

var dat map[string]interface{}

if err := json.Unmarshal(b, &dat); err != nil { 
   panic(err)     
}    
fmt.Println(dat) 

earnings := dat["earnings"].(float64)     
fmt.Println(earnings)

See this code on playground:

https://play.golang.org/p/y_l8MttsOH6

What if I have a large file and I want only a few fields and avoid decoding all of it?

json.RawMessage is a raw encoded JSON value. It can be used to delay JSON decoding.

We can unmarshal the JSON data into a map[string]json.RawMessage and pick out the fields we want.

E.g. In the JSON, below, I want to get the values of only two fields, accountid and paymentid and ignore the rest.

{
     "accountid": "162",
     "name": "myclient",
     "paymentid": "A345345",
     "sequence": 3977318850,
     "paymentrow": [{
         "date": "2018 - Jul - 22",
         "amount ": 137.18
     }, {
         "date": "2018-Jul-22",
         "amount": 137.18
     }]
 }

This is how I can get the two fields I want:

var objmap map[string]*json.RawMessage

err = json.Unmarshal(jsonData, &objmap)
if err != nil {
    fmt.Println(err)
}

var accountID, paymentID string 
err = json.Unmarshal(*objmap["accountid"], &accountID) 
if err != nil {
  fmt.Println(err)
}

err = json.Unmarshal(*objmap["paymentid"], &paymentID)
if err != nil {
   fmt.Println(err)
}

https://play.golang.org/p/62WpFGBxZEa