GSoC Code Submission

Hey guys. I read the guidance on submitting your work for final review and it said the best way was to write a blog post to detail how and what I did. I know my mentors are going to read this but I'm writing it as if many people will. 

Where do I start, in England we would say I've ridden the hills, meaning there's been many ups and downs but I have managed to still plough on. I think it's closer to what I would think riding the grand canyon would be like, some days were real bad, and others were amazing. I came into GSoC thinking I knew quite a bit about the open source world but I quickly noticed I had no idea :) I just thought I'd bullet point out what I have learnt so far:

• Use Git and IRC - I've said it loads of times and I will probably say it a few more :) I can see how this has already benefited me and how it will benefit me in the future.
• Just generally having the confidence to speak up - every week I share my progress and opinion on what I should do next.
• learning about coding conventions and stuff I thought I already knew - just as an example I had no idea what most of the acronyms meant within the code base. Once someone said what they meant, I had the greatest epiphany ever. CLI (command line interface) stumped me for ages. I feel like it's worth talking about because I know there are people out in the world who have had a similar path through their computer science education and will have missed out on the stuff I did. 

Issue #286

Now getting to what I actually completed during GSoC. I was set on the task during my community bonding period to fix a small issue to get me used to doing the stuff I was going to be doing all summer. I was a  mess. My first commit was absolutely terrifying because I had never posted my code online before. I didn't know who was going to read it or how bad the feedback might have been. nevertheless, this commit was the result of HOURS of work reading into stdin and looking into the dav1d codebase. 

Turns out the input file is first probed for a header and then read again to get the bulk of the data. I learnt the hard way that you cant just jump around in stdin and this made my life hell. I'm not going to lie, I felt really bad for ages about the comments I got on my first pull request because I did get a working version on my laptop. However, the people who reviewed my code said it wouldn't work on other platforms and I was using bad practices anyway. On top of all of that, the commit messages did not follow the correct format and looked a bit clunky.

The two following commits [1][2] were me trying to relearn Git the correct way. I had no idea that when you changed branch, the actual local workspace on my laptop changed. When I look back, I think how did I think any different, but it just shows I've come a long way.

Just talking about the issues I tried to solve doesn't really do the work I did justice. I messed around and dug around the encoder and decoder just to see how they worked too.For example, I did some test encodings with multiple AV1 encoders. I have learnt a great deal throughout the GSoC summer, things like how different encoders and decoders have different design goals such as SVT-AV1 and Rav1e. I also made some synthetic test sequences and it was quite fun comparing the different the metadata for each type of sequence, natural or synthetic was. One of my supervisors, Fredrik had a 'Russian companies metadata extractor' and I gathered some sequences together, shipped them to him and asked him to give me the metadata back. I would say this was my highest and most successful learning method. I could compare different metrics and alter my synthetic test sequences to see if I could change a specific metric. Boring stuff I know, but its actually addictive :) The test sequences I made can be found here.

AreWeCompressedYet? - Nope, I'm not reciting Chewbacca and Han's conversation while they were in the trash compactor on Vodran :) I'm talking about the super cool online platform to run sequence encodings on! I have a bit of a soft spot for the platform and I have used both arewecompressedyet and internal platforms while interning with other companies. I see limitations with the online platform, and the biggest problem with making a codec is having it adopted. Not many people are that bothered about the whole codec world, as long as its acceptable, they take what they get instead of seeking out the best performance. I feel that having a distributed compression platform in the form of a downloadable client will engage people with the community and widen the impact of designing a codec will have. I know this sounds like another GSoC proposal, it's not, but I don't start term till October and I feel as if I have taken from GSoC more than I gave back. I feel once I have finished with my review of GSoC, I would like to kick start a solution I can contribute to during the next few months.  

CLI and implementing --metadata

This went through some changes throughout my GSoC summer. Originally I came up with a solution so that specifying --metadata would inhibit the output of the actual decoding and only output what I wanted to output. My commits can be found here. This turned out to slow me down tremendously because of the 'leakage' of the actual decoded stream would still output where I hadn't stopped it fully.

I was having issues with the way I was outputting stuff. I changed the ‘--debug metadata’ cli option to ‘--metadata <file name>’ so I can output to a different file to the actual output. The work is here. This includes error messages and usage info. I also looked into having a config file to enable and disable the metadata that would get pulled out of the decoder but this turned out to be too complicated because I had never done anything like that before. I think this was the right decision and to do nothing around this idea probably saved a lot of time. 

I then changed the metadata output file so that each frame had it's own file and the argument given in the cli would be the folder directory of the metadata files. This is the commit. Doing this showed me how to get arguments from the command line and introduced me to get_opt. At the time of writing the commit, I thought this was unique to the dav1d code base but actually it is a standard library kind of. I learnt about how I can ask the code to look for a specific letter like '-m' with or without an argument by specifying ':'. I know it doesn't sound that good, but when you have a line of code that sets a variable to  "i:o:m:vql:s:" you just have to look into every aspect of how it works.

Those annoying pipelines

Maybe it was just me who really hated the pipelines but I found it really hard to actually pass the building stages. I had to commit every time I wanted to retest if my code would pass the pipelines. This is what the bulk of my commits are 'fixing pipeline issues', you can see a lot of them on this page. 

Given hindsight I should have read the committing guidance before contributing and then I wouldn't have had to spend more time rebaseing than actually coding :)

I had issues with building on other OS. something I had never come across before. Linux was a new thing for me and learning bash was a hard step in my journey. One night at 3 a.m. I was on call with my supervisor learning how to commit and iron out my misconceptions surrounding Git - even though I didn't know how to commit code at that time, I knew I had to commit myself  :) It scared me how quickly my supervisor could ssh into my computer and show me first hand how to use bash, even though he was 20,000 miles away. It got the job done though! In true open source spirit I made a document about how to use git so that everyone who passes by my page will be slightly better than they were before.

My three big Issues

Communication - It nice having so many people who want to see me succeed and want to know about my progress but I find it hard to keep them all up to date with what I am doing. I assume if I tell one of my supervisors, they will talk about me at some point and hear what they need to hear. This wasn't the case because I got the feeling that my day-to-day supervisor Fredrik didn't really talk about me to my other supervisors and I only found this out 2 weeks ago. I could have done better with communicating and if anything GSoC has shown me why regular communication is important and not just a big dump of information every now and again. Say that though, I did attend every meeting apart from one and I shared my progress very clearly in each of them.

 

Lack of understanding of codec processes  - It's hard looking for some key values to extract from the codebase if you don't know what the variables mean in the first place. I think this is what I will take most away from this summer, learning the key processes in the encoding and decoding processes, the key tools and all the acronyms associated with them. The community was so nice and one guy Luca did a mini boot-camp into the encoding processes and this helped more than I was expecting. I feel when going into the subject of video, people assume you know stuff when actually I didn't, 'It is very hard to get into video' - Unlord 2020 :)

University work - I came off a placement year early due to COVID-19. I knew there was some admin stuff to do during the summer break but I didn't know the scale of writing I had to do. It felt as if I was writing 5000 words a week just to have the words 'with placement' on my degree.

Not forgetting about my well documented journey :)

I have written 3 blog posts throughout my GSoC summer which I feel give a very good insight into both my work but also the physical and emotion ups and downs I have had to face too. These can be found here [1][2].

Its not all been plain sailing though, the cornavirus has took its toll in many ways. I feel like I have been trapped in my parents house with nowhere to go and sat at my desk for eternity. It hasn't just effected GSoC for the students, but also for the supervisors, I feel like the community has had a great big hit of losses.

Did I follow my proposal and complete my goals?

One issue with my proposal was it was quite vague to begin with. I somehow had a week by week rundown of what I was going to do and also have it be quite vague. My first moth was community binding, I wish I had spoke out in the community more than I did in this period but I can say I was pleased with the effort into reading I had. I read for a substantial length of time familiarising myself with the language Rust and completed a handful of programming challenges to prove to myself I was fluent enough to continue. My rust knowledge was only beginner at best but I never used it anyways.

Also in my community bonding month I studied the current analysis tools within libaom and learnt what analysis was being done at the time. This proved to be a tricky one because I had found that both mindfreeze and a guy called Kyle (brilliant bloke) were also doing something similar, and that's not to mention the aom inspect tool that already existed. Nevertheless, I plodded on and I eventually moved over to the forums after signing onto the mailing lists. I spent a few days just reading the conversations and the archives.

My supervisor recommended I look at issue #286 which was to add stdin support for dav1d. I spent so long on this to the point where I was done with it. I think it wasn't the task itself but rather the sudden shock of realising I didn't know how to use git and understand common coding practices. Nevertheless I had still met my proposal criteria of 'dabbling in the codebase' I guess. The key commits for this have already been referenced above but they are [1][2][3]

In the screenshot above, you can see the major floor in my design. It would check if the user wanted to input from stdin and then seek back to the start of the standard input after probing it. 

The aim of my first month was to make some synthetic test sequences and find some natural ones too. I then encoded them and treated them as my 'unit tests' and studied how if I change them, how did the metadata change. The metadata at this point was not my own, but rather supplied by my supervisor Fredrik. The screenshot below is an example frame of my synthetic test sequences and shows how they moved over time. The other screen shot is the metadata my supervisor shared with me in hope that my outcome was something similar.

By the second month I was outputting my own metadata in JSON format and I had committed a quite a lot more than I was expecting. I had issues with pipelines and keeping up with the daily emails but I was still sticking to my success criteria. The one thing I never did in my second month was solicit feedback from the community about my project. I can now see how asking the community's advice would have lead to a much greater adoption of my program. 

The metadata was originally just me writing to a file within the david codebase wherever I saw a piece of metadata I needed. This became clutered and unmaintainable so I had to look at other methods. I was told to drag all the data structures I needed to access into a new file with headers to link everything together. This meant I could just call my functions within this file and the david codebase didnt have to see any of my code. To implement this I needed to learn how to link files together with header files and how to modify the make files to include my code in the compilation process. Below is a screenshot of my metadata header file. It only allows access to the non static functions within my file and hides the other complex functions away from the users.

I had to make a piece of code that outputted the data into JSON format and this was harder said than done. below is are several functions that allow me to write elements of JSON into a file. allow I have to do is call one of these functions and it writes a bit of JSON for me.

There are exceptions, I needed a way of writing sets of data into the files. I had to make some more functions which would write curly braces at the right indentation level and write the metadata in the set correctly. Below are the functions I wrote to allow me to write sets:

A behemoth of a task was to follow, I had to output the metadata I had chosen using my functions I had made. I cannot show the code in detail because there is quite a lot of code to show and it is also quite repetitive. The code can be found here.

The third month flew by, I had deadlines for university, my birthday and coronavirus struck hard on me and the others I worked with. Nevertheless I kept to my schedule and extracted more metadata. Sure enough, the screenshot below shows I was able to output was my supervisor outputted using his 'Russian metadata extractor' and I am pretty proud of it. 

At the time of writing this, some of the outputted metadata is being outputted as int rather than u_int16 and therefore looks a really negative or really positive in value. I have a pull request ongoing and I am fixing this issue while writing this. 

Pull request can be found here: https://code.videolan.org/videolan/dav1d/-/merge_requests/1072

Conclusions

I would just like to say a great thanks to Unlord, JB, Mindfreeze, Luca and Fredrik, without them my GSoC summer wouldn't have been what it was. My GSoC journey has been great and I have been able to experience the opensource community and what it's like to program with professionals in there field. I think GSoC has been more of a learning experience for me, and the stipend has enabled me to get experience in my career path rather than working in retail. This stipend will allow me to continue to explore my career path before starting university again in October. 

Thanks again guys,

Tom.