Golang

Bit manipulation in Go

Bit manipulation is important to know when doing data compression, cryptography and optimization.

Bitwise operations include AND, NOT, OR, XOR and bit shifts.

Go supported bit operations include:

& AND
| OR
^ XOR
&^ AND NOT
<< left-shift
>> right-shift

Go does not have a dedicated NOT operator like C++ or Python. Instead we have to use the XOR operator to toggle the bits.

var n byte = 0x0F  
fmt.Printf("%08b\n", n)
n = ^n
fmt.Printf("%08b\n", n)

This is the output:

00001111
11110000

Here are some common bit manipulation algorithms that can be done in Go

  • Swapping integers using XOR since XOR will result in 1 only if both bits are the same.
func swap(a, b int) (int, int) {   
a = a ^ b
b = b ^ a
a = a ^ b
return a, b
}
  • Toggle bits using XOR
func flip(a int) int{  
a ^= 0xFFFFFFFF
return a
}
  • Find Odd/even using AND
func isEven(n int) bool{  
if n&1 == 1 {
return false
}
return true
}
  • Find if a number is power of 2
func isPowerofTwo(n int) bool{  
if (n & (n-1) == 0){
return true
}
return false
}
  • Left  Shift to multiply by 2
func multiplyBy2(uint num) uint{
return num << 1
}
  • Right Shift to divide by 2
func divideBy2(uint num) uint{
return num >> 1
}
  • Add without using addition operator
func add(a, b int64) int64 {
    for b != 0 {

        // common bits of a and b go to carry
        carry := a & b

        // xor - sum bits where one is not set
        a = a ^ b

        // shift carry by 1
        b = carry << 1
    }

    return a
}

Go has a built in package that has functions to manipulate bits

https://golang.org/pkg/math/bits/

Golang

Reflecting on structs

Reflection in Go allows  to manipulate objects and are most useful when dealing with structs.

I always wished I could range over a struct. But “range” only supports builtin types such as string, list, and map. Reflect makes it easy to do so.

I have a package that contains a function, I use frequently to do exactly this.  It takes in a struct and copies over the elements to a map so we can range over it.  https://github.com/mariadesouza/structutil

Here is a small example:

package main
import ( "fmt"
 "github.com/mariadesouza/structutil"
)
type testStruct struct { 
   Name       string 
   Email      []string 
   Occupation string
}
func main() {
  test := testStruct{"Ethan", []string{"emdesouza@gmail.com"}, "engineer"} 
  m := structutil.StructToMap(&test) 

  for key, val := range *m { 
    fmt.Println(key, ":", val)
  }
}

How it works?

Main concepts used:

type

Every variable in GO has a static type including elements of a struct.

interface{}

In go an interface{} represents an empty set of methods that satisfies all variable types

reflect

Go has a package called reflect that allows us to inspect the type and value stored in any data type.

I used reflect on a struct  and turn it into a map of names to values.  The call to ValueOf returns a Value representing the run-time data. This helps us to loop through the runtime elements

structmap := make(map[string]interface{}) // map to hold key - element
 elements := reflect.ValueOf(myStruct).Elem()  
 typeofT := elements.Type()    // type of Element
 for i := 0; i <elements.NumField(); i++ { 
     f := elements.Field(i) 
     elementName := typeofT.Field(i).Name //key name of element 
     valueOfElement := f.Interface() //returns current value as an interface{}
     structmap[elementName] = valueOfElement 
 } 

For this to work,  the struct elements have to be exported i.e. start with a capital or you will get a panic error as below.

panic: reflect.Value.Interface: cannot return value obtained from unexported field or method

https://play.golang.org/p/Kyn-k4rAFFX

Access private struct fields using reflection

We can read private struct variables in a package using reflection. However,  private field values cannot be changed.

E.g.

package main
import (
	"fmt"
	"reflect"
	"container/list"
)

func main() {
	cl := list.New()
        cl.PushFront(2)
	fieldvaluelen := reflect.ValueOf(cl).Elem().FieldByName("len")
	fmt.Println(fieldvaluelen.Int()) 
}

Output: 1

Reference

Laws of reflection

Golang

Mocking with Golang

Overview

Writing unit tests are vital and ensures the integrity of your code. But sometimes it is  hard to write tests for code that references external dependencies. That is where Go makes mocking such services easy.

The core idea is to create the interface with one or more of the methods from the original package struct.

For example, I wrote a wrapper SFTP package  to create SFTP connections and download/upload files.

https://github.com/mariadesouza/sftphelper

Inorder to test this package, rather than downloading and uploading files to an actual SFTP server, I used  mocking so it would still test the flow of the package.

Implementation

First define an interface so we can mock the sftphelper.SFTPConnection struct:

type SftpConnector interface {
	RemoveFile(string) error
}

Write the mock struct using the interface

type FakeSftpConnector struct {
	
}
// make sure it satisfies the interface
var _ SftpConnector = (*FakeSftpConnector)(nil)

func (f *FakeSftpConnector) RemoveFile(source string) error {
...
}

Update your function that takes the sftphelper.SFTPConnection struct function to take the new interface instead

In your test code, you can send in the fake struct instead and make assertions on the inputs after calling the target function

func TestProcessInnerFiles(t *testing.T) {
	tt := &FakeSftpConnector{}
	err := processInnerFiles(tt, "/home/ftptest/feed_20160912/", "IN7994_20160715.json")
	if err != nil {
		t.Errorf("Error was not expected in processInnerFiles: %s", err)
	}
}

 References

 

Golang

Custom Unmarshal JSON in Go

While defining a struct for reading JSON text, we normally define a one to one relationship between the JSON field and the struct variable.

I start by converting the JSON to a Golang struct. My favorite JSON to Go converter is: https://mholt.github.io/json-to-go/

The JSON can be unmarshalled into that struct directly using the encoding/json package in the stdlib. Note that the struct has to have exported fields defined within it i.e. capitalized field names.

https://golang.org/pkg/encoding/json/

Here is an example to unmarshal JSON data that is well defined.

type Response struct {     
Page int `json:"page"`
Entries []string `json:"entries"`
}

str := `{"page": 1, "entries": ["classA", "classB"]}`

res := response{}
json.Unmarshal([]byte(str), &res)

See it in action: https://play.golang.org/p/7IEki4G7AEA

However, not all JSON data is well defined.

What do we do if we have to unmarshal JSON data that does not completely adhere to the struct we define?

We could custom unmarshal this JSON data into a struct that we define by writing our own custom unmarshal function.

E.g. The crypto currency exchange GDAX market feed API.

This API has an endpoint get-product-order-book that returns JSON data that gives us bids and asks for a crypto currency. These values are returned as array within an array with three floats and the fields are not well defined.

https://docs.gdax.com/?javascript#get-product-order-book

Values returned are:

{ 
"sequence": 3977318850, 
"bids": [ 
          [ "4625.78", "0.80766325", 3 ] 
        ], 
"asks": [ 
          [ "4625.79", "3.0154341", 3 ] 
        ]
}

The data within the array actually represents the following three values:

[price, size, order_id]

The JSON converter gave me this golang struct:

type AutoGenerated struct { 
Sequence int64 `json:"sequence"`
Bids [][]interface{} `json:"bids"`
Asks [][]interface{} `json:"asks"`
}

Ideally, I would like to define the struct as below and Unmarshal the API contents directly into this struct.

type Bid struct {     
Price     string
Size      string
NumOrders int
}
type OrderBook struct {  
Sequence int64 `json:"sequence"`  
Bids     []Bid `json:"bids"`  
Asks     []Bid `json:"asks"`
}

But the Unmarshaller will not automatically unmarshal into the above defined struct.

E.g. The Unmarshal code below using the struct, throws this error : json: cannot unmarshal array into Go value of Bid

body, err := ioutil.ReadAll(resp.Body)  
if err != nil {  
fmt.Println(err)
}
var orders OrderBook
err = json.Unmarshal(body, &orders)
if err != nil {
fmt.Println(err)

Unmarshal calls the UnmarshalJSON method of the value. Hence, for the above conversion, we can define a custom UnmarshalJSON function  for our array to struct conversion. Since we want the inner struct Bid to be unmarshalled correctly the method has to be defined on that struct.

func (b *Bid) UnmarshalJSON(data []byte) error {     

var v []interface{}
if err := json.Unmarshal(data, &v); err != nil {
return err
}

b.Price, _ = v[0].(string)
b.Size, _ = v[1].(string)
b.NumOrders = int(v[2].(float64))

return nil
}

We Unmarshal the data into an interface and then we can set the values as desired with a type assertion.

This will cause the  Unmarshal on the main struct to work seamlessly!

The full code is available on github at https://github.com/mariadesouza/go-orderbook.

Partial Unmarshal of JSON

At times you want to read a JSON file and extract just a few fields.

If you want to avoid defining a struct, you could decode it into a map of generic types and cast them appropriately.

b := []byte(`{"earnings":1000.50,"names":["Joe","Jane"]}`)

var dat map[string]interface{}

if err := json.Unmarshal(b, &dat); err != nil {
panic(err)
}
fmt.Println(dat)

earnings := dat["earnings"].(float64)
fmt.Println(earnings)

See this code on playground:

https://play.golang.org/p/y_l8MttsOH6

What if I have a large file and I want only a few fields and avoid decoding all of it?

json.RawMessage is a raw encoded JSON value. It can be used to delay JSON decoding.

We can unmarshal the JSON data into a map[string]json.RawMessage and pick out the fields we want.

E.g. In the JSON, below, I want to get the values of only two fields, accountid and paymentid and ignore the rest.

{
"accountid": "162",
"name": "myclient",
"paymentid": "A345345",
"sequence": 3977318850,
"paymentrow": [{
"date": "2018 - Jul - 22",
"amount ": 137.18
}, {
"date": "2018-Jul-22",
"amount": 137.18
}]
}

This is how I can get the two fields I want:

var objmap map[string]*json.RawMessage

err = json.Unmarshal(jsonData, &objmap)
if err != nil {
fmt.Println(err)
}

var accountID, paymentID string
err = json.Unmarshal(*objmap["accountid"], &accountID)
if err != nil {
fmt.Println(err)
}

err = json.Unmarshal(*objmap["paymentid"], &paymentID)
if err != nil {
fmt.Println(err)
}

https://play.golang.org/p/62WpFGBxZEa

Golang · Uncategorized

Golang Primer

Golang is simple, efficient and has potential to revolutionize  backend processing. It is the optimal language for microservices  architecture.

The best way to get started with Golang is to read the official document.

https://golang.org/doc/

IDEs for Go

I have used Sublime, Atom and Visual Studio and even plain old Emacs for writing Go code. Of all the IDEs, I found Visual Studio ideal for me as a developer learning a new language as well as to quickly navigate through projects built together with my team. For more on using Visual Studio with Go, read the link below.

https://code.visualstudio.com/docs/languages/go

Best Practices

  • Prior to all checkins please run gofmt -w yourfile.go (IDEs have plugins that run this for you)
  • Packages are given lower case, single-word names; there should be no need for underscores or mixedCaps
  • Interface names and variables Convention in Go is to use MixedCaps or mixedCaps rather than underscores to write multiword names
  • Only functions / variables starting with a capital letter would be usable outside the package. All other internal functions start with lower case
  • Panics should as much as possible not cross package boundaries and if they do it should be made clear in the package documentation.
  • There is linter tool https://github.com/golang/lint which recommends “proper” syntax for your programs (cd myProject; golint ./… )
  • Other great tools:
    • errcheck
    • go vet
    • go test -race

Programming in Go

Although Go borrows from a lot of languages, it has some characteristics that are either left out or unusually implemented. For the most part it is done to achieve a more simplistic syntax and to write more efficient code. It is important to know these idioms especially for experienced programmers that have written code in other languages.

For a comprehensive detail on the Go language the best resource is the official Golang document.

https://golang.org/doc/effective_go.html

I have listed below a few nuances of the language that stood out to me when developing with Go:

  • There is no do or while loop in Go. You can use a generalized for;
  • There is no automatic fall through in switch cases in Go.
  • We can use break to terminate a switch or loop early. In addition a label can be used to break out of a nested loop. u
  • Use defer for closing handles and unlocking mutexes immediately after opening or locking. This will ensure you don’t forget to do so later.
  • We can return the address of a local variable created within a function. This can be used outside the function. E.g.
    • return &Node{Data: d, Next: nil}
  • make is used only for slices, channels and maps
  • new is a rarely used in Go. new(T) allocates storage for T and returns its address .
  • Go’s arrays and slices are one-dimensional. To create two dimensional arrays or slices, we define it as an array of arrays or a slice of slice of slices. An inner slice can be of different lengths.
  • Only interfaces can be embedded within interfaces.

Deployment

Manage Go package dependencies using godep:

https://github.com/tools/godep

Building for Linux on mac

Setup Cross compilation One needs to compile Go compiler for different target platforms and architectures. This is done from src folder in go installation. If you use Homebrew on OS X, then you have a simpler solution, install go with cross compilers:

 $ brew install go --with-cc-all  

Then to build:

GOOS=linux GOARCH=386 CGO_ENABLED=0 go build -o appname-linux appname.go

Optimize Go binaries in production

These options will help shrink Go binaries sizes by removing some debugging data. Note that panics will still give you info.

-s Omit the symbol table and debug information.

-w Omit the DWARF symbol table.

Go Gotchas

This is a compilation of example that I encountered in Go that either stumped me or I had to find a workaround for. Making a list helps me remember them when I’m coding.

No built-in race condition safety

Go does not restrict access to shared data. A Go programmer has to explicitly synchronize and manages shared resources, else there could be race conditions making for some messy real-time problems.

You can use Go’s race detector to check a race condition in your Go code:

go run -race incrementor.go

Note that the race detector has a limit of 8192 simultaneously alive goroutines. Reduce the number of goroutines in a program if you want to verify it with the race detector.

Date Formatting

Go just doesn’t use an abstract syntax for datetime components (YYYY-MM-DD), but these exact numbers. No other number will do!

These are predefined layouts for use in Time.Format and Time.Parse. The reference time used in the layouts is the specific time: Mon Jan 2 15:04:05 MST 2006 which is Unix time 1136239445.

Since MST is GMT-0700, the reference time can be thought of as 01/02 03:04:05PM ’06 -0700 To define your own format, write down what the reference time would look like formatted your way; see the values of constants like ANSIC, StampMicro or Kitchen for examples.

The model is to demonstrate what the reference time looks like so that the Format and Parse methods can apply the same transformation to a general time value.

Marshal UTF-8 characters to JSON

Marshaling a string like “TV & Home Entertainment”, will generate JSON like

 "category": "TV \\u0026 Home Entertainment",

The best way to do this is to disable HTML escaping as follows.

buf := new(bytes.Buffer) 
enc := json.NewEncoder(buf)
enc.SetEscapeHTML(false)
enc.Encode(&textToBeMarshalled)
fmt.Println(buf.String())

Maps and struct

Assign a struct field off map has to be done by first creating the element in golang.

type apiNode struct { 
method, expectedfile string
}
var handlerMap map[string]apiNode
handlerMap = make(map[string]apiNode)
handlerMap[APIName] = apiNode{"GET","a.json"}

Reference Links

Godoc to the source – https://blog.golang.org/godoc-documenting-go-code