Manish Posted on May 31 Learning, Experimenting - Concurrency in Go # go # concurrency Torn between writing a toy project I'd use vs learning Go, I decided to try my hand at writing a simple distributed chunked filestore. Here are a few things I learned and how I moved from a naive implementation to a parallelized implementation. There are a lot of filestore papers already that involve much more sophisticated distribution/replication strategies, and I will be building upon my project as I continue - I've got lots to learn. Anyhow, let's get started! The Test Suite First up, a simple test suite: type fakeReader struct { remaining int } func ( r * fakeReader ) Read ( p [] byte ) ( int , error ) { if r . remaining <= 0 { return 0 , io . EOF } n := len ( p ) if n > r . remaining { n = r . remaining } for i := 0 ; i < n ; i ++ { p [ i ] = 'x' } r . remaining -= n return n , nil } func BenchmarkStoreChunk ( b * testing . B ) { const size = 1000 << 20 s := & Server { config : Config { Store : b . TempDir (), Chunksize : 10 , MaxConcurrency : 1 }} b . ResetTimer () b . SetBytes ( int64 ( size )) for b . Loop (){ r := & fakeReader { remaining : size } if _ , err := s . storeChunks ( r , "bench" ); err != nil { b . Fatal ( err ) } } } Enter fullscreen mode Exit fullscreen mode Pretty straightforward. Initialize a 1000 MB byte array, call my main storeChunks function to handle the write to disk. Note that Chunksize (in MB) and MaxConcurrency (num of concurrent workers) is a configurable parameter at server initialization (for the parallelized implementation). Running it is simply: go test - bench =. - benchmem - count = 3 Enter fullscreen mode Exit fullscreen mode The option count parameter runs it 3 times in this example. Note - I will be skipping the server handling and the input read part, and will be going over the core storage logic only. 1. The Naive Write numChunks := 0 chunkSize := 5 << 20 buf := make ([] byte , chunkSize ) for { part , err
LIVE