Golang

Packages on the Go

Package management with Go is a very talked about issue. Unfortunately, go get does not support functionality to fetch specific tags or versions. It gets the package from the HEAD in git.

Recently we had a situation where a developer on our team used a package and then it got obsoleted. The package developer tagged the release before making breaking changes and the sources were still available but we had to do a git checkout and run it.

That is when I looked into package management. Go 1.5 introduced the “vendor” directory as an experiment and made it official  in Go 1.6

If you use third party packages in your product, copy it to the vendor directory and go searches for dependencies there.

Ex.

package main 

import (
   "fmt"
   "io"

   "https://github.com/mariadesouza/sftphelper" 
)

 

main.go
vendor
 |--github.com
 |     |--mariadesouza
 |     |    |-- sftphelper
 |     |    |     |-- LICENSE
 |     |    |     |-- README.md 
 |     |    |     |--sftphelper.go
Golang

Pass by value or reference

The official Go site FAQ states,  “As in all languages in the C family, everything in Go is passed by value”. This is because the function gets a copy of everything that is passed in.

Is there such thing as pass by reference in Go?

There are different views  as to what is exactly pass by reference to Go.  Some strongly maintain there is no such thing as pass by reference. In C++ terms, the actual meaning of pass by reference is you pass a reference or a pointer to the actual data structure rather the data itself. The function then can modify the value of the argument using that reference.

In Go when I pass a pointer to a struct for example, whether its a copy to the pointer or not, I am not passing the struct itself but a pointer or a reference to it.  I can modify the actual struct using the pointer. In my view, that fits the definition of pass by reference.

When to pass a pointer?

We don’t need to pass pointers to map and slices  as they are already descriptors that contain pointers to the actual map or slice data.

Compelling arguments to use pointer receiver and pass by reference:

  • You want to modify the receiver. With value receivers you can’t modify the struct itself
  • Its is a big struct. It will cost to deep copy the struct.

When you pass a slice to a function, since it is a pointer to an array you just get a copy of the slice structure. It will still point to the same underlying array segment. So any modifications made to the slice within the function will be seen outside.

https://play.golang.org/p/LrrHtK86WmC

However, if you append an element, remember that a new slice is created and elements are copied over so you will lose the elements if this happens within a function so you must return a slice. E.g. append from the stdlib returns a new slice.

Whatever it is you choose to use, stay consistent. Coming from a C++ background, I always thought pass by reference is cheaper than passing by value. But apparently it is not always the case in Go.

Interesting Reads:

http://goinbigdata.com/golang-pass-by-pointer-vs-pass-by-value/

Golang

Bit manipulation in Go

Bit manipulation is important to know when doing data compression, cryptography and optimization.

Bitwise operations include AND, NOT, OR, XOR and bit shifts.

Go supported bit operations include:

& AND
| OR
^ XOR
&^ AND NOT
<< left-shift
>> right-shift

Go does not have a dedicated NOT operator like C++ or Python. Instead we have to use the XOR operator to toggle the bits.

var n byte = 0x0F  
fmt.Printf("%08b\n", n)
n = ^n
fmt.Printf("%08b\n", n)

This is the output:

00001111
11110000

Here are some common bit manipulation algorithms that can be done in Go

  • Swapping integers using XOR since XOR will result in 1 only if both bits are the same.
func swap(a, b int) (int, int) {   
a = a ^ b
b = b ^ a
a = a ^ b
return a, b
}
  • Toggle bits using XOR
func flip(a int) int{  
a ^= 0xFFFFFFFF
return a
}
  • Find Odd/even using AND
func isEven(n int) bool{  
if n&1 == 1 {
return false
}
return true
}
  • Find if a number is power of 2
func isPowerofTwo(n int) bool{  
if (n & (n-1) == 0){
return true
}
return false
}
  • Left  Shift to multiply by 2
func multiplyBy2(uint num) uint{
return num << 1
}
  • Right Shift to divide by 2
func divideBy2(uint num) uint{
return num >> 1
}
  • Add without using addition operator
func add(a, b int64) int64 {
    for b != 0 {

        // common bits of a and b go to carry
        carry := a & b

        // xor - sum bits where one is not set
        a = a ^ b

        // shift carry by 1
        b = carry << 1
    }

    return a
}

Go has a built in package that has functions to manipulate bits

https://golang.org/pkg/math/bits/

Golang

Reflecting on structs

Reflection in Go allows  to manipulate objects and are most useful when dealing with structs.

I always wished I could range over a struct. But “range” only supports builtin types such as string, list, and map. Reflect makes it easy to do so.

I have a package that contains a function, I use frequently to do exactly this.  It takes in a struct and copies over the elements to a map so we can range over it.  https://github.com/mariadesouza/structutil

Here is a small example:

package main
import ( "fmt"
 "github.com/mariadesouza/structutil"
)
type testStruct struct { 
   Name       string 
   Email      []string 
   Occupation string
}
func main() {
  test := testStruct{"Ethan", []string{"emdesouza@gmail.com"}, "engineer"} 
  m := structutil.StructToMap(&test) 

  for key, val := range *m { 
    fmt.Println(key, ":", val)
  }
}

How it works?

Main concepts used:

type

Every variable in GO has a static type including elements of a struct.

interface{}

In go an interface{} represents an empty set of methods that satisfies all variable types

reflect

Go has a package called reflect that allows us to inspect the type and value stored in any data type.

I used reflect on a struct  and turn it into a map of names to values.  The call to ValueOf returns a Value representing the run-time data. This helps us to loop through the runtime elements

structmap := make(map[string]interface{}) // map to hold key - element
 elements := reflect.ValueOf(myStruct).Elem()  
 typeofT := elements.Type()    // type of Element
 for i := 0; i <elements.NumField(); i++ { 
     f := elements.Field(i) 
     elementName := typeofT.Field(i).Name //key name of element 
     valueOfElement := f.Interface() //returns current value as an interface{}
     structmap[elementName] = valueOfElement 
 } 

For this to work,  the struct elements have to be exported i.e. start with a capital or you will get a panic error as below.

panic: reflect.Value.Interface: cannot return value obtained from unexported field or method

https://play.golang.org/p/Kyn-k4rAFFX

Access private struct fields using reflection

We can read private struct variables in a package using reflection. However,  private field values cannot be changed.

E.g.

package main
import (
	"fmt"
	"reflect"
	"container/list"
)

func main() {
	cl := list.New()
        cl.PushFront(2)
	fieldvaluelen := reflect.ValueOf(cl).Elem().FieldByName("len")
	fmt.Println(fieldvaluelen.Int()) 
}

Output: 1

Reference

Laws of reflection

Golang

Mocking with Golang

Overview

Writing unit tests are vital and ensures the integrity of your code. But sometimes it is  hard to write tests for code that references external dependencies. That is where Go makes mocking such services easy.

The core idea is to create the interface with one or more of the methods from the original package struct.

For example, I wrote a wrapper SFTP package  to create SFTP connections and download/upload files.

https://github.com/mariadesouza/sftphelper

Inorder to test this package, rather than downloading and uploading files to an actual SFTP server, I used  mocking so it would still test the flow of the package.

Implementation

First define an interface so we can mock the sftphelper.SFTPConnection struct:

type SftpConnector interface {
	RemoveFile(string) error
}

Write the mock struct using the interface

type FakeSftpConnector struct {
	
}
// make sure it satisfies the interface
var _ SftpConnector = (*FakeSftpConnector)(nil)

func (f *FakeSftpConnector) RemoveFile(source string) error {
...
}

Update your function that takes the sftphelper.SFTPConnection struct function to take the new interface instead

In your test code, you can send in the fake struct instead and make assertions on the inputs after calling the target function

func TestProcessInnerFiles(t *testing.T) {
	tt := &FakeSftpConnector{}
	err := processInnerFiles(tt, "/home/ftptest/feed_20160912/", "IN7994_20160715.json")
	if err != nil {
		t.Errorf("Error was not expected in processInnerFiles: %s", err)
	}
}

 References

 

Golang

Custom Unmarshal JSON in Go

While defining a struct for reading JSON text, we normally define a one to one relationship between the JSON field and the struct variable.

I start by converting the JSON to a Golang struct. My favorite JSON to Go converter is: https://mholt.github.io/json-to-go/

The JSON can be unmarshalled into that struct directly using the encoding/json package in the stdlib. Note that the struct has to have exported fields defined within it i.e. capitalized field names.

https://golang.org/pkg/encoding/json/

Here is an example to unmarshal JSON data that is well defined.

type Response struct {     
Page int `json:"page"`
Entries []string `json:"entries"`
}

str := `{"page": 1, "entries": ["classA", "classB"]}`

res := response{}
json.Unmarshal([]byte(str), &res)

See it in action: https://play.golang.org/p/7IEki4G7AEA

However, not all JSON data is well defined.

What do we do if we have to unmarshal JSON data that does not completely adhere to the struct we define?

We could custom unmarshal this JSON data into a struct that we define by writing our own custom unmarshal function.

E.g. The crypto currency exchange GDAX market feed API.

This API has an endpoint get-product-order-book that returns JSON data that gives us bids and asks for a crypto currency. These values are returned as array within an array with three floats and the fields are not well defined.

https://docs.gdax.com/?javascript#get-product-order-book

Values returned are:

{ 
"sequence": 3977318850, 
"bids": [ 
          [ "4625.78", "0.80766325", 3 ] 
        ], 
"asks": [ 
          [ "4625.79", "3.0154341", 3 ] 
        ]
}

The data within the array actually represents the following three values:

[price, size, order_id]

The JSON converter gave me this golang struct:

type AutoGenerated struct { 
Sequence int64 `json:"sequence"`
Bids [][]interface{} `json:"bids"`
Asks [][]interface{} `json:"asks"`
}

Ideally, I would like to define the struct as below and Unmarshal the API contents directly into this struct.

type Bid struct {     
Price     string
Size      string
NumOrders int
}
type OrderBook struct {  
Sequence int64 `json:"sequence"`  
Bids     []Bid `json:"bids"`  
Asks     []Bid `json:"asks"`
}

But the Unmarshaller will not automatically unmarshal into the above defined struct.

E.g. The Unmarshal code below using the struct, throws this error : json: cannot unmarshal array into Go value of Bid

body, err := ioutil.ReadAll(resp.Body)  
if err != nil {  
fmt.Println(err)
}
var orders OrderBook
err = json.Unmarshal(body, &orders)
if err != nil {
fmt.Println(err)

Unmarshal calls the UnmarshalJSON method of the value. Hence, for the above conversion, we can define a custom UnmarshalJSON function  for our array to struct conversion. Since we want the inner struct Bid to be unmarshalled correctly the method has to be defined on that struct.

func (b *Bid) UnmarshalJSON(data []byte) error {     

var v []interface{}
if err := json.Unmarshal(data, &v); err != nil {
return err
}

b.Price, _ = v[0].(string)
b.Size, _ = v[1].(string)
b.NumOrders = int(v[2].(float64))

return nil
}

We Unmarshal the data into an interface and then we can set the values as desired with a type assertion.

This will cause the  Unmarshal on the main struct to work seamlessly!

The full code is available on github at https://github.com/mariadesouza/go-orderbook.

Partial Unmarshal of JSON

At times you want to read a JSON file and extract just a few fields.

If you want to avoid defining a struct, you could decode it into a map of generic types and cast them appropriately.

b := []byte(`{"earnings":1000.50,"names":["Joe","Jane"]}`)

var dat map[string]interface{}

if err := json.Unmarshal(b, &dat); err != nil {
panic(err)
}
fmt.Println(dat)

earnings := dat["earnings"].(float64)
fmt.Println(earnings)

See this code on playground:

https://play.golang.org/p/y_l8MttsOH6

What if I have a large file and I want only a few fields and avoid decoding all of it?

json.RawMessage is a raw encoded JSON value. It can be used to delay JSON decoding.

We can unmarshal the JSON data into a map[string]json.RawMessage and pick out the fields we want.

E.g. In the JSON, below, I want to get the values of only two fields, accountid and paymentid and ignore the rest.

{
"accountid": "162",
"name": "myclient",
"paymentid": "A345345",
"sequence": 3977318850,
"paymentrow": [{
"date": "2018 - Jul - 22",
"amount ": 137.18
}, {
"date": "2018-Jul-22",
"amount": 137.18
}]
}

This is how I can get the two fields I want:

var objmap map[string]*json.RawMessage

err = json.Unmarshal(jsonData, &objmap)
if err != nil {
fmt.Println(err)
}

var accountID, paymentID string
err = json.Unmarshal(*objmap["accountid"], &accountID)
if err != nil {
fmt.Println(err)
}

err = json.Unmarshal(*objmap["paymentid"], &paymentID)
if err != nil {
fmt.Println(err)
}

https://play.golang.org/p/62WpFGBxZEa

Uncategorized

Go Data Types

Basic data types

bool 
string
int int8 int16 int32 int64
uint uint8 uint16 uint32 uint64 uintptr
byte // same as for uint8
rune // represents a Unicode code point
float32 float64
complex64 complex128

int

int is a signed integer type that is at least 32 bits in size. It is a distinct type, however, and not an alias for, say, int32

An int is either 32 or 64 bits depending on the platform. So it maybe better to be specific if developing and host machines differ.

Rune

Strings are made up of “runes” rather than chars. Rather than having char and wchar for characters and unicode characters, a rune is a “code point” as defined in the Unicode standard. Internally it is  a 32-bit representation.

nihao := "你好" 
for index, runeValue := range nihao {
fmt.Printf("Char %#U Byte position %d\n", runeValue, index)
}
  • String to array of runes
runes := []rune(s)
  • rune array to string
str := string(runes)

Simple Composite Types

Slice

A slice consists of a pointer to an array segment, length of the segment and its capacity i.e. the maximum length of the segment.

  • Create
b = make([]T, len(a))

Note that since the zero value of a slice is nil, you don’t need to explicitly make a slice, you can just declare it like below.

  • Add element
var s []int      // s := make([]string, 0) 
s = append(s, 1) //add one element
s = append(s, 2,3,4) //add multiple
s = append(s, s1…) // append another slice
  • Delete element
s = s[:len(s)-1] //remove last element

Be aware that if a slice contains pointers to elements and a delete is done, there maybe a potential memory leak. Values referenced by the removed slice elements may not be garbage collected as the references are not totally removed.

  • Remove all elements of a slice

To remove all elements of a slice, set it to nil. This will release it to the garbage collector as long as there no other references to it.

s = nil
  • Iterate over a slice
for index, value := range s { 
fmt.Println(index, value)
}
  • Copy

Use copy to deep copy all slice elements from one slice to another. Simply assigning one slice will only change the reference.

a := []int{1, 2}
b := []int{3, 4}

a = b // changes a to refer to the location of b

copy(a,b) //actually copy values from b to a

https://play.golang.org/p/X_WC1eN7IHCh

The number of elements copied is the minimum of len(src) and len(dst). For example in the below code the destination slice will have no elements since len(dst) is 0

s := []int{4, 2,3, 1}
var dst []int
copy(dst, s)
fmt.Println(dst)
fmt.Println(s)

Output:
[]
[4 2 3 1]

We need to allocate a length when creating the destination slice

dst := make([]int, len(s)
copy(dst, s)

https://play.golang.org/p/Sp4NoX4YaaA

Two dimensional slices

Slices in Go are one-dimensional. We have to compose it to form two-dimensional arrays.

matrix := make([][]int, row) 
for m := range matrix {
matrix[m] = make([]int, col)
}

When you pass a slice to a function, since it is a pointer to an array you just get a copy of the slice structure. It will still point to the same underlying array segment. So any modifications made to the slice within the function will be seen outside.

https://play.golang.org/p/OLkoxGf8uaZ

However, if you append an element, remember that a new slice is created and elements are copied over so you will lose the elements if this happens within a function so you must return a slice. E.g. append from the stdlib returns a new slice.

  • Creating and initializing a 2d slice
graph := [][]int{  
{0, 4, 0, 0, 0},
{4, 0, 8, 0, 0},
{0, 8, 0, 7, 0},
{0, 0, 7, 0, 9},
{0, 0, 0, 9, 0},
}

Map

The built-in map type implements a hash table.


Maps, like channels, but unlike slices, are just pointers to runtime types.

Map is a pointer to a runtime.hmap structure. So when we declare a map using var, its value will be nil until initialized using make. Attempts to access a nil map will result in a panic.

Maps are not concurrent safe. So proceed with caution when reading/writing to them in more than one go routine.

  • Create
x := make(map[int]string)

You could initialize a map with values. In that case, you don’t have to use make

colorCodes := map[string]string{  
"BRONZE":   "E0001B",
"GOLD":    "A89968",
"SILVER":   "888B8D",
}
fmt.Println(colorCodes["BRONZE"])
  • Add element to a map
x[1] = 10
  • Delete element
delete(x, 1)
  • Find value
val, ok := x[key]
  • Iterating
for key, val:= range x {     
println("Key:", key, "Value:", val ,"Value:", m[k])
}
  • Size
len(m)
  • Copy

To copy a map to another, we need to traverse the map and do a deep copy. Just assigning a map to another will not copy contents over.

https://play.golang.org/p/JiuMeE3rClf

All operations on a map are in constant time. Most map operations like lookup, insertion etc. are rewritten at compile time to runtime calls. 

You don’t need to use references with a map. Map types are reference types like slices.  If you pass a map to a function and add key-value pairs or update values , it will be reflected outside a function. When using maps in concurrent routines, we need to add locks to ensure data integrity.

https://play.golang.org/p/j7dD1K_rfRY

Struct Type

struct is a collection of fields.

A struct can have methods defined on it. There are two kinds of methods that can be defined on a  struct, a value receiver and a pointer receiver.

type T struct { 
a int
}

func (tv T) Mv(a int) int {
return 0
} // value receiver

func (tp *T) Mp(f float32) float32 {
return 1
} // pointer receiver

It all comes down which one is better for the problem at hand.

Related blog posts for Structs:

Constants

There are boolean constantsrune constantsinteger constantsfloating-point constantscomplex constants, and string constants. Rune, integer, floating-point, and complex constants are collectively called numeric constants.

A constant expression may contain only constant operands and are evaluated at compile time. As such, there is no such thing as array constant in Go. Slices and arrays are always evaluated during runtime. They can only be defined as variables.

Auto increment constants can be defined using iota.

const (
Black = iota
White
Red
Yellow
Green
)

fmt.Println(Black, White, Red, Yellow, Green)

Output:
0 1 2 3 4

Equality of data types

Simple data types and individual elements can be compared using the equality operator ==

Arrays and slices can only be compared to nil.

For other complex datatypes we can use the DeepEqual function in the reflect package.

package main
import (
"fmt"
"reflect"
)

func main(){
a := []int{3,4,5}
b := []int{3,4}
b = append(b, 5)
if reflect.DeepEqual(a,b){
fmt.Println("slices a and b are equal")
}
}

Struct values can also be compared using DeepEqual. The exported as well as unexported values are compared for equality.

Channels can be compared like numbers and strings using the == operator.

Related:

Interface Types