Uncategorized

Channels and Workerpools

Concurrency are part of the Golang core. They are similar to light weight threads. We run a routine using the go keyword.

go matchRecordsWithDb(record)

Channels

Channels are a way to synchronize and communicate with go routines.

ch := make(chan string) 
ch <- "test" //Send data to unbuffered channel
v := <-ch //receive data from channel and assign to v

The receive here will block till data is available on the channel.

Most programs use multiple go routines and  buffered channels.

doneCh := make(chan bool, 4)

Here we will be able to run a routine 4 times without blocking.

select

select is like a switch with cases that allows you to wait on multiple communication operations. It is a way to receive channel data will block until one of its case is ready.  Select with a default clause is a way to implement non-blocking sends, receives.

select {
case ac <- a:
// sent a on ac
case b:= <-bc:
// received b from bc
}

WorkerPools

I encountered the classic scenario where I had to make thousands of database calls to match records from a file to database entries. Finding viable matches in the database per line in the file, was slow and proving to much of a hassle. I wanted to add concurrency to my calls to achieve this faster. However, I was restricted by the database connection. I could only send a set number of queries at a time or it would error out.

I started with the naive approach. I  create a buffered channel of 100 that went out and call the matching routine. The requirement was to match with a key in a table and return results.  Some improvement. It did about 100 queries. Wait for those to finish and start the next batch of 100.

const workers = 100 
jobsCh := make(chan int, workers)
for rowIdx := 0; rowIdx < len(fileRecords); rowIdx += workers {
for j = 0; j < workers; j++ {
if (rowIndex + j) >=len(fileRecords) {
break;
}
go matchRecordsWithDb(jobsCh, &fileRecords[rowIdx+j])
} // wait for the 100 workers to return
for i := 0; i < j; i++ {
fmt.Printf("%d", <-jobsCh)
}
}

There was a major snag in this approach. We had a condition, that if the line in the file didn’t have the main id field, we had to query on another field. This field was not indexed and took a while to query.  It is a very old legacy system so I can’t change the indexing at this point.

In my entire file I had one such record. The iteration of the 100 workers that had among it the routine to do this one query waited almost a minute and a half on that one query, while the 99 others finished. That is when I started looking at design patterns with channels and came across worker pools.

Worker pools is an approach to concurrency in which a fixed number of m workers have to do n number of  tasks in a work queue. Rather than wait on all the workers (channels) at once, as the workers get idle they can be assigned jobs.

The three main players are :

Collector:  Gathers all the jobs

AvailableWorkers Pool: Buffered channel of channels that is used to process the requests

Dispatcher: Pulls work requests off the Collector and sends them to available channels

All the jobs are add to a collector pool. The dispatcher picks jobs off the collector. If there are availableWorkers it gives them the job else it tries to createOne. If all m workers are busy doing jobs the dispatcher will wait on completion to assign the job.

After reading and experimenting with workerPools, I have written a workerPool package that can be used directly or as a guideline to implement your own.

https://github.com/mariadesouza/workerpool

Uncategorized

Microservices with gRPC

A microservice is an independent runnable services that does one task effectively.  The concept is rather than having one monolithic application, we break it up into independent services that can be easily maintained.

To effectively use microservices there has to be a way for the various independent services to communicate with each other.

There are two ways of communication between microservices:

1. REST, such as JSON or XML over http
2. gRPC – Lightweight RPC protocol brought out by Google

What is gRPC?

To understand gRPC we first take a look at RPC.

RPC(Remote Procedure Call) is a form of inter-process communication (IPC), in that different processes have different address spaces. RPC is a kind of request–response protocol. RPC enables data exchange and invocation of functionality residing in a different address space or process.

gRPC is based around the idea of defining a service, specifying the methods that can be called remotely with their parameters and return types. A client application can call methods on a server application as if it were a local object.

  • gRPC uses the new HTTP 2.0 spec
  • It allows for bidirectional streaming
  • It uses binary rather than text and that helps keep the payload compact and efficient.

This is Google’s announcement for gRPC.

So whats the “g” in gRPC? Google? Per the official FAQ page, gRPC stands for  gRPC Remote Procedure Calls i.e. it is a recursive acronym.

Protocol Buffers

gRPC uses protocol buffers as Interface Definition Language (IDL) for describing both the service interface and the structure of the payload messages.

Protocol buffers are a mechanism for serializing structured data. Define how you want your data to be structured once, then you can use special generated source code to easily write and read your structured data to and from a variety of data streams and using a variety of languages.

https://developers.google.com/protocol-buffers/docs/overview

Specify how you want the information you’re serializing to be structured by defining protocol buffer message types in .proto files. This message is encoded to the protocol buffer binary format.

message Person {
  required string name = 1;
  required int32 id = 2;
  optional string email = 3;
}

gRPC in Go

go get -u google.golang.org/grpc
go get -u github.com/golang/protobuf/protoc-gen-go

protobuf.Protobuf allows you to define an interface to your service using a developer friendly format.

Uncategorized

String manipulation in Go

Basic Operations

  • Get char array of a String
greeting := "Comment allez-vous" 
greeingCharacterArr := []rune(greeting)
  • Get a char at the specific index
fmt.Printf("%c", greeting[5])
fmt.Println(greeting[5])

The result would be  “n“. However, without the character formatter, the result would be the Unicode decimal code 110.

  • Get string length
len(greeting)
  • Substrings
func substring(s string, beginIndex int) string{
return s[beginIndex:]
}

func substring2(s string, beginIdx int, endIdx int) string {
return s[beginIdx:endIdx]
}
  • String to Integer
testN,_ := strconv.Atoi("1234")  
testN += 1
fmt.Println(testN)

strconv package

Above examples : https://play.golang.org/p/kds_Lu9HyTJ

  • Comparing Strings

To compare strings, use the comparison operators ==, < and >

The strings package has a Compare function that comes with a warning that no one should use strings.Compare

https://golang.org/src/strings/compare.go

Sorting and Searching Strings in Golang

The sort package in Go has some very useful methods for sorting and searching slices and strings.

The sort.StringSlice attaches the methods of Interface to []string, sorting in increasing order. https://golang.org/pkg/sort/#StringSlice

E.g.

package main 
import (
"fmt"
"sort"
)

func main() {
countries := sort.StringSlice{"USA", "India", "South Africa"}
countries.Sort()
n := countries.Search( "USA")
fmt.Println(""Result: ", n, countries[n])

}


Result: 3 USA

Sorting Characters in a string

To sort characters in a string, we use the fact that strings in Go can be represented as a slice of runes. We could extend the generic sort function in Go to sort the slice of runes. To use the generic sort, we need to implement sort.Interface – Len, Swap, Less.

type sortRunes []rune 

func (s sortRunes) Less(i, j int) bool {
return s[i] < s[j]
}

func (s sortRunes) Len() int{
return len(s)
}

func (s sortRunes) Swap(i, j int) {
s[i], s[j] = s[j], s[i]
}

str1 := "sortthisstring"
s := []rune(str1) sort.Sort(sortRunes(s))

See the full working example here:

https://play.golang.org/p/f7wFD8NX97d

Note the sort package comments, Sort makes one call to data.Len to determine n, and O(n*log(n)) calls to   data.Less and data.Swap. 

Uncategorized

Embedding in Go

Go supports defining is-a relationships using an embedded type.

Fields and methods in a struct have a has-a relationship

E.g.  Person has-a name and email.

type Person struct { 
Name string
Email string
}

Embedding to define an is-a relationships  is declared with a type but no explicit field name. The unqualified type name acts as the field name.

type Employee struct {        
Person
EmployeeID string
}

This way the Person struct can be accessed using the type name:

accountant := new(Employee) 
accountant.Person.Pay()

We can also invoke any Person methods directly on the Employee object

accountant := new(Employee) 
accountant.Pay()

We cannot embed a slice or a map

type Employee struct {   
[]Person
EmployeeID string
}

This will give  a syntax error: unexpected [, expecting field name or embedded type.

A workaround will be defining a type and then embedding :

type SpecialPeople []Person 
type Employee struct {
SpecialPeople
EmployeeID string
}

The spec in Go describes embedding as below:

EmbeddedField = [ "*" ] TypeName .

Embed by-pointer

The advantage of embedding by reference is that  you are embedding all the functionality of a type without needing to know when it is instantiated. The major application of this would be to have thousands of instances sharing a single underlying data structure. This can significantly reduce memory consumption.

type Image struct {      
data [5][5]
}
type Block struct {
*Image
show bool
}

Because, the unqualified type name acts as the field name for an embedded struct, we can’t have an embedded struct and its pointer in the same struct.

type Image struct { 
T // conflicts with embedded field *T and *P.T
*T // conflicts with embedded field T and *P.T
*P.T // conflicts with embedded field T and *T
}

Promotion of fields or methods

All field or method calls for embedded type objects are resolved at compile-time without the use of a virtual table. A field or method of an embedded field in a struct is called promoted.

type Person struct {        
Name   string
Email  string
}
func (p *Person) FreeGift() int64{
return 100
}
type Employee struct {      
Person      
EmployeeID string 
}

In this example person.FreeGift()  and employee.FreeGift() is the same so the method is promoted. Promotion  occurs only at the first level.

https://play.golang.org/p/N8rXyIR_CW

However, if we have a method with the same name defined on the parent struct as below, then employee.FreeGift() will invoke the method on the Employee struct.

func (e *Employee) FreeGift() int64{        
return 200
}

In the above example, calling person.FreeGift() and employee.Person.FreeGift() will invoke the method on the Person struct

https://play.golang.org/p/lYUTijVF7U

One very important design aspect to remember is, if you plan to use an embedded struct and the embedded type has non-exported fields or methods,  those are completely inaccessible to you in a separate package.

Embedding Interfaces

Embedding an interface will add all (exported and non-exported) methods of the embedded interface to the enclosing interface.

type ImageWriter interface {        
Read(b Buffer) bool
Write(b Buffer) bool
}
type ImageFile interface {
ImageWriter  // same as adding methods of ImageWriter
Close()
}

Check out this example to add logging to your struct by embedding the log.Logger object.

https://play.golang.org/p/orTWK_TaM_W

Uncategorized

Go Data Types

Basic data types

bool 
string
int int8 int16 int32 int64
uint uint8 uint16 uint32 uint64 uintptr
byte // same as for uint8
rune // represents a Unicode code point
float32 float64
complex64 complex128

int

int is a signed integer type that is at least 32 bits in size. It is a distinct type, however, and not an alias for, say, int32

An int is either 32 or 64 bits depending on the platform. So it maybe better to be specific if developing and host machines differ.

Rune

Strings are made up of “runes” rather than chars. Rather than having char and wchar for characters and unicode characters, a rune is a “code point” as defined in the Unicode standard. Internally it is  a 32-bit representation.

nihao := "你好" 
for index, runeValue := range nihao {
fmt.Printf("Char %#U Byte position %d\n", runeValue, index)
}
  • String to array of runes
runes := []rune(s)
  • rune array to string
str := string(runes)

Simple Composite Types

Slice

A slice consists of a pointer to an array segment, length of the segment and its capacity i.e. the maximum length of the segment.

  • Create
b = make([]T, len(a))

Note that since the zero value of a slice is nil, you don’t need to explicitly make a slice, you can just declare it like below.

  • Add element
var s []int      // s := make([]string, 0) 
s = append(s, 1) //add one element
s = append(s, 2,3,4) //add multiple
s = append(s, s1…) // append another slice
  • Delete element
s = s[:len(s)-1] //remove last element

Be aware that if a slice contains pointers to elements and a delete is done, there maybe a potential memory leak. Values referenced by the removed slice elements may not be garbage collected as the references are not totally removed.

  • Remove all elements of a slice

To remove all elements of a slice, set it to nil. This will release it to the garbage collector as long as there no other references to it.

s = nil
  • Iterate over a slice
for index, value := range s { 
fmt.Println(index, value)
}
  • Copy

Use copy to deep copy all slice elements from one slice to another. Simply assigning one slice will only change the reference.

a := []int{1, 2}
b := []int{3, 4}

a = b // changes a to refer to the location of b

copy(a,b) //actually copy values from b to a

https://play.golang.org/p/X_WC1eN7IHCh

The number of elements copied is the minimum of len(src) and len(dst). For example in the below code the destination slice will have no elements since len(dst) is 0

s := []int{4, 2,3, 1}
var dst []int
copy(dst, s)
fmt.Println(dst)
fmt.Println(s)

Output:
[]
[4 2 3 1]

We need to allocate a length when creating the destination slice

dst := make([]int, len(s)
copy(dst, s)

https://play.golang.org/p/Sp4NoX4YaaA

Two dimensional slices

Slices in Go are one-dimensional. We have to compose it to form two-dimensional arrays.

matrix := make([][]int, row) 
for m := range matrix {
matrix[m] = make([]int, col)
}

When you pass a slice to a function, since it is a pointer to an array you just get a copy of the slice structure. It will still point to the same underlying array segment. So any modifications made to the slice within the function will be seen outside.

https://play.golang.org/p/OLkoxGf8uaZ

However, if you append an element, remember that a new slice is created and elements are copied over so you will lose the elements if this happens within a function so you must return a slice. E.g. append from the stdlib returns a new slice.

  • Creating and initializing a 2d slice
graph := [][]int{  
{0, 4, 0, 0, 0},
{4, 0, 8, 0, 0},
{0, 8, 0, 7, 0},
{0, 0, 7, 0, 9},
{0, 0, 0, 9, 0},
}

Map

The built-in map type implements a hash table.


Maps, like channels, but unlike slices, are just pointers to runtime types.

Map is a pointer to a runtime.hmap structure. So when we declare a map using var, its value will be nil until initialized using make. Attempts to access a nil map will result in a panic.

Maps are not concurrent safe. So proceed with caution when reading/writing to them in more than one go routine.

  • Create
x := make(map[int]string)

You could initialize a map with values. In that case, you don’t have to use make

colorCodes := map[string]string{  
"BRONZE":   "E0001B",
"GOLD":    "A89968",
"SILVER":   "888B8D",
}
fmt.Println(colorCodes["BRONZE"])
  • Add element to a map
x[1] = 10
  • Delete element
delete(x, 1)
  • Find value
val, ok := x[key]
  • Iterating
for key, val:= range x {     
println("Key:", key, "Value:", val ,"Value:", m[k])
}
  • Size
len(m)
  • Copy

To copy a map to another, we need to traverse the map and do a deep copy. Just assigning a map to another will not copy contents over.

https://play.golang.org/p/JiuMeE3rClf

All operations on a map are in constant time. Most map operations like lookup, insertion etc. are rewritten at compile time to runtime calls. 

You don’t need to use references with a map. Map types are reference types like slices.  If you pass a map to a function and add key-value pairs or update values , it will be reflected outside a function. When using maps in concurrent routines, we need to add locks to ensure data integrity.

https://play.golang.org/p/j7dD1K_rfRY

Struct Type

struct is a collection of fields.

A struct can have methods defined on it. There are two kinds of methods that can be defined on a  struct, a value receiver and a pointer receiver.

type T struct { 
a int
}

func (tv T) Mv(a int) int {
return 0
} // value receiver

func (tp *T) Mp(f float32) float32 {
return 1
} // pointer receiver

It all comes down which one is better for the problem at hand.

Related blog posts for Structs:

Constants

There are boolean constantsrune constantsinteger constantsfloating-point constantscomplex constants, and string constants. Rune, integer, floating-point, and complex constants are collectively called numeric constants.

A constant expression may contain only constant operands and are evaluated at compile time. As such, there is no such thing as array constant in Go. Slices and arrays are always evaluated during runtime. They can only be defined as variables.

Auto increment constants can be defined using iota.

const (
Black = iota
White
Red
Yellow
Green
)

fmt.Println(Black, White, Red, Yellow, Green)

Output:
0 1 2 3 4

Equality of data types

Simple data types and individual elements can be compared using the equality operator ==

Arrays and slices can only be compared to nil.

For other complex datatypes we can use the DeepEqual function in the reflect package.

package main
import (
"fmt"
"reflect"
)

func main(){
a := []int{3,4,5}
b := []int{3,4}
b = append(b, 5)
if reflect.DeepEqual(a,b){
fmt.Println("slices a and b are equal")
}
}

Struct values can also be compared using DeepEqual. The exported as well as unexported values are compared for equality.

Channels can be compared like numbers and strings using the == operator.

Related:

Interface Types

Golang · Uncategorized

Golang Primer

Golang is simple, efficient and has potential to revolutionize  backend processing. It is the optimal language for microservices  architecture.

The best way to get started with Golang is to read the official document.

https://golang.org/doc/

IDEs for Go

I have used Sublime, Atom and Visual Studio and even plain old Emacs for writing Go code. Of all the IDEs, I found Visual Studio ideal for me as a developer learning a new language as well as to quickly navigate through projects built together with my team. For more on using Visual Studio with Go, read the link below.

https://code.visualstudio.com/docs/languages/go

Best Practices

  • Prior to all checkins please run gofmt -w yourfile.go (IDEs have plugins that run this for you)
  • Packages are given lower case, single-word names; there should be no need for underscores or mixedCaps
  • Interface names and variables Convention in Go is to use MixedCaps or mixedCaps rather than underscores to write multiword names
  • Only functions / variables starting with a capital letter would be usable outside the package. All other internal functions start with lower case
  • Panics should as much as possible not cross package boundaries and if they do it should be made clear in the package documentation.
  • There is linter tool https://github.com/golang/lint which recommends “proper” syntax for your programs (cd myProject; golint ./… )
  • Other great tools:
    • errcheck
    • go vet
    • go test -race

Programming in Go

Although Go borrows from a lot of languages, it has some characteristics that are either left out or unusually implemented. For the most part it is done to achieve a more simplistic syntax and to write more efficient code. It is important to know these idioms especially for experienced programmers that have written code in other languages.

For a comprehensive detail on the Go language the best resource is the official Golang document.

https://golang.org/doc/effective_go.html

I have listed below a few nuances of the language that stood out to me when developing with Go:

  • There is no do or while loop in Go. You can use a generalized for;
  • There is no automatic fall through in switch cases in Go.
  • We can use break to terminate a switch or loop early. In addition a label can be used to break out of a nested loop. u
  • Use defer for closing handles and unlocking mutexes immediately after opening or locking. This will ensure you don’t forget to do so later.
  • We can return the address of a local variable created within a function. This can be used outside the function. E.g.
    • return &Node{Data: d, Next: nil}
  • make is used only for slices, channels and maps
  • new is a rarely used in Go. new(T) allocates storage for T and returns its address .
  • Go’s arrays and slices are one-dimensional. To create two dimensional arrays or slices, we define it as an array of arrays or a slice of slice of slices. An inner slice can be of different lengths.
  • Only interfaces can be embedded within interfaces.

Deployment

Manage Go package dependencies using godep:

https://github.com/tools/godep

Building for Linux on mac

Setup Cross compilation One needs to compile Go compiler for different target platforms and architectures. This is done from src folder in go installation. If you use Homebrew on OS X, then you have a simpler solution, install go with cross compilers:

 $ brew install go --with-cc-all  

Then to build:

GOOS=linux GOARCH=386 CGO_ENABLED=0 go build -o appname-linux appname.go

Optimize Go binaries in production

These options will help shrink Go binaries sizes by removing some debugging data. Note that panics will still give you info.

-s Omit the symbol table and debug information.

-w Omit the DWARF symbol table.

Go Gotchas

This is a compilation of example that I encountered in Go that either stumped me or I had to find a workaround for. Making a list helps me remember them when I’m coding.

No built-in race condition safety

Go does not restrict access to shared data. A Go programmer has to explicitly synchronize and manages shared resources, else there could be race conditions making for some messy real-time problems.

You can use Go’s race detector to check a race condition in your Go code:

go run -race incrementor.go

Note that the race detector has a limit of 8192 simultaneously alive goroutines. Reduce the number of goroutines in a program if you want to verify it with the race detector.

Date Formatting

Go just doesn’t use an abstract syntax for datetime components (YYYY-MM-DD), but these exact numbers. No other number will do!

These are predefined layouts for use in Time.Format and Time.Parse. The reference time used in the layouts is the specific time: Mon Jan 2 15:04:05 MST 2006 which is Unix time 1136239445.

Since MST is GMT-0700, the reference time can be thought of as 01/02 03:04:05PM ’06 -0700 To define your own format, write down what the reference time would look like formatted your way; see the values of constants like ANSIC, StampMicro or Kitchen for examples.

The model is to demonstrate what the reference time looks like so that the Format and Parse methods can apply the same transformation to a general time value.

Marshal UTF-8 characters to JSON

Marshaling a string like “TV & Home Entertainment”, will generate JSON like

 "category": "TV \\u0026 Home Entertainment",

The best way to do this is to disable HTML escaping as follows.

buf := new(bytes.Buffer) 
enc := json.NewEncoder(buf)
enc.SetEscapeHTML(false)
enc.Encode(&textToBeMarshalled)
fmt.Println(buf.String())

Maps and struct

Assign a struct field off map has to be done by first creating the element in golang.

type apiNode struct { 
method, expectedfile string
}
var handlerMap map[string]apiNode
handlerMap = make(map[string]apiNode)
handlerMap[APIName] = apiNode{"GET","a.json"}

Reference Links

Godoc to the source – https://blog.golang.org/godoc-documenting-go-code