I have tried out reinforcement learning with the frozen lake example. In this post I will introduce the concept of Q-learning, the TensorFlow Agents library, the Frozen Lake game environment and how I put it all together.

When I play board games, I cannot stop myself from seeing the abstract rules, building a decision tree in my head and wondering how one could play an optimal round. This has led me to write a couple articles about board games. The one about Scythe features an implementation of parts of the game and the attempt to find a good game using just beam search. That was a first step, but I want to do better.

Read more…

Wir hangeln uns ja aktuell von einem »Lockdown« zum nächsten. Dazwischen dann mit Lockerungen und dann wieder einem »Lockdown«. Auf Twitter habe ich was schönes gesehen, das wollte ich hier einmal reproduzieren. Wenn man eine feste Grenze für die Inzidenz hat, so wie aktuell die 100, dann führt das nicht zu einem Ende vom Lockdown.

In einem ganz einfachen Modell habe ich bei einer Inzidenz von 90 gestartet und eine Steigerung von 10 % jeden Tag angenommen, die es keinen »Lockdown« gibt. Und sobald das Mittel der letzten drei Tage über 100 ist, kommt der »Lockdown« und die Inzidenz nimmt jeden Tag um 10 % ab. Damit erhält man für die nächsten 30 Tage dann diesen Verlauf:

Wir gehen also zwischen »Lockdown« und Lockerungen hin und her, und es kommt alles zyklisch. Mit dieser Regelung alleine wird man die Pandemie nicht regulieren können.

Read more…

My university background is physics, which is an empirical science. All measurements or derived quantities must be quoted with an error estimate. This should ideally include both a statistical and a systematic error. If you take a look at the paper from my thesis, you will find this table:

It contains the comparison of a certain quantity (pion scattering length) from various other publications and our result. There are uncertainties given, either as a combined value or as separate statistical and systematic ones. One result by the ETMC even has asymmetric systematic errors quotes. You can look at these numbers and figure out whether the result from that work lies within the error budget of the other works. The raw table isn't as meaningful as a plot, but you could create a plot from the table. And one can see that our result has a one standard deviation confidence interval of $[-0.0567, -0.0395]$, so it easily encompasses all previous work. Also it has the largest error of all results, so it is the least precise addition to the field. This is okay as it was just an auxiliary result and we weren't aiming for precision there.

In my new field, natural language processing (NLP), I cannot say the same thing. There are no error estimates whatsoever! And it really annoys me, you cannot really derive any conclusion from the data. Take for instance the paper on BERT. They show a table where they compare two variants of the BERT model with other language models in various tasks:

Read more…

In Deutschland gibt es jetzt ein neues Energielabel (siehe tagesschau.de). Das löst das alte Label ab, hat aber letztlich die gleichen Kategorien wie vorher.

Als die damals das Label eingeführt hatten, fand ich das wirklich großartig. So können Verbraucher recht einfach sehen, wie effizient das Gerät mit Ressourcen umgeht. Zugleich sind die Hersteller dann im Zugzwang ihre Geräte effizienter als die der Konkurrenz zu machen, wenn sie weiter verkaufen wollen.

In der Implementierung haben sie dann allerdings sehr kurzfristig agiert, genauso wie die Grafikkartenhersteller. Jenen fällt nämlich immer nach 10 Generationen auf, dass ihr Nummernschema zu Ende ist. So habe ich bei NVIDIA erlebt, wie sie 8x00 und dann 9x00 hatten. Sie haben ja vier Ziffern, aber nutzen nur die erste für die Generation, und die zweite für die Abstufung. Die restlichen zwei danach sind sinnlose Nullen. Und danach haben sie dann die 1x0 rausgebracht, die 2x0 bis zur 9x0. Ja, da war es dann wieder zuende. Und dann haben sie die 10x0 rausgebracht. Naiv hoffte ich auf zweistellige Generationen, aber sie haben die 20x0 rausgebracht. Aktuell sind wir bei 30x0, also in ein paar Jahren haben wir dann 90x0 und NVIDIA wird wieder überrascht sein.

Read more…

Ich kann normalerweise ganz gut hören, ob irgendwo noch ein Fenster geöffnet ist. Der Straßenlärm klingt dann ganz anders. Mit geschlossenen Fenstern ist es dann deutlich dumpfer. In meiner neuen Wohnung habe ich aber häufig den Eindruck, dass die Fenster nicht ganz geschlossen waren. Da es sich um Plastikfenster handelt, kann man das eigentlich nicht falsch machen. Aber vorbeifahrende Autos sind irgendwie immer so präsent …

Irgendwann habe ich mir mal die Fenster genauer angeschaut. Und da habe ich nicht schlecht gestaunt, als ich auf das Konzept Fensterfalzlüfter gestoßen bin. Das ganze ist ein Konzept zur kontinuierlichen Raumbelüftung. Das Fenster hat zwei Dichtungen, einmal im Rahmen und einmal im Flügel. Die Dichtung im Rahmen ist unten an der Seite unterbrochen, sodass dort kalte Luft von außen dort einströmen kann.

Diese wird erwärmt und steigt dann zwischen der inneren und äußerden Dichtung auf. Sie kommt oben an, dort ist der Fensterfalzlüfter eingebaut. Letztlich ist das nur ein Plastikteil mit einer kleinen Klappe. Sollte es zu sehr ziehen, kann sie den Spalt somit schließen.

Read more…

Scythe is a board game from the engine builder genre. Is a rather complicated game as there are different types of entities and actions around. It is about collecting resources, deploying the hero, workers, mechs and buildings, moving them around, fighting with others, accomplishing achievements. In the end one obtains a score based on the number of achievements, controlled hexes and resources as well as left-over money and some bonus points.

The player will pick a random player mat, the following shows the innovative mat (German version):

Every turn one picks a new column and performs the top, bottom or both actions. There are certain patterns that lead to successful game play. Most importantly one should strive to do both the top and bottom action in order to get the most done each turn. With the innovative board, the bolster action is combined with the build action. Therefore one should have some wood ready when picking this to perform both actions. But wood is collected either through the trade or produce top-row actions, which have their own bottom-row actions. Different player mats have different arrangements, therefore the cycles that one goes through are also different. After playing this game a couple of times, one gets better at building a good engine.

Whenever I play such games, I see a game graph in my head. Abstract models seem to be more present than the beautiful artwork of the game. So I spend a whole rainy Saturday actually implementing a subset of the game and let if figure out good ways to build that engine. In this blog post I want to share the code, explain my approach and share some results. There is so much more to be done, but there is only so much one can do in one sitting.

Read more…

I've defended my PhD thesis on 2020-11-26. After that I needed to publish my thesis in order to receive my PhD diploma. This is more complicated than it sounds.

Decades ago, when one did not have servers with PDF files, one had to publish the thesis with a lot of hard copies. And this unfortunately is still the default mode at the University of Bonn. So there are a bunch of hoops that one has to take in order to have the thesis published.

I took a couple days before I started this process. On 2020-12-01 I sent the PDF file via e-mail to the university library. They have a special address where I should sent it to. As the PDF file was so large, I have just uploaded it to my website and gave them the link.

In parallel I needed to register my thesis on some other platform of the library. There they did not accept the PDF, just the meta data. And the form had a “Submit” and a “Reset” button there. I mean, how many times does one actually want to reset a form? I thought that this “Reset” button died out decades ago. After registering my thesis there, I had to download a receipt, sign it and send it via e-mail to the university library.

On the website of the library they told me that I needed to make sure that my advisor is okay with me publish my thesis electronically. They did not mention any particular form. So I just asked my advisor and it was okay.

Read more…

Recently I have moved and since I've basically exchanged flats with somebody else and her friend, I have been driving around three locations for a whole weekend. I would try to figure out which stuff to load into the car, where to drive next, which person to bring along on the free seat. One would of course need to drive the car where the stuff should go to, but also have a sufficient number of people to carry stuff and put up furniture.

This has reminded me of the classical salad-sheep-fox riddle, which goes like this: You have a salad, a sheep and a fox. They are all on one bank of a river and you want to cross it. You have a raft which can only hold one thing at a time. But you cannot leave the sheep unattended with the fox, otherwise the fox will kill the sheep. Yet you cannot leave the salad unattended with the sheep, as the sheep will eat it. You need to figure out a way to transfer the stuff without leaving something unattended.

So you cannot start ferrying the salad, the fox will kill the sheep. You cannot start ferrying the fox, the sheep will eat the salad. You will need to ferry the sheep first. Then you ferry back. We can take either the salad or the fox. Let us try the salad. We end up with the salad and the sheep on the other bank, the fox at the start. We cannot just leave them there together unattended. But ferrying back the salad makes no sense, so we ferry back the sheep. On the starting bank we exchange the sheep for the fox, ferry over the fox. We then got the salad and the fox together on the other bank, the sheep left behind. We sail back, load in the sheep, and ferry it to the other bank. Done.

This should be solvable systematically. We have three objects, and three places where they can be. This makes for 27 states, but we can already rule out the states where more than one element is on the raft. This gives us the following states, with the original river bank on the right and the other bank on the left. The raft is in the middle.

Read more…

I have finished the text my PhD thesis at the end of August 2020. This meant that the scientific and writing part was done and from there on out it would be just formalities that I had to do. Of course there still was the defense, but first I needed to do a bunch of things in order to get there.

First one has to print the thesis and bind it. The examination office required me to hand in five copies. Four for the members of my examination board and then another one for the office. The binding and cardboard costs like 10 EUR per copy. That was the easy part.

I had to put in a formal request for an examination. For that I needed a to have an examination panel of four professors. One needs to have a second examiner from theoretical physics, a third examiner from experimental physics and a forth one outside of physics. The three people I have asked happily agreed to be on my panel and I was very glad that it went so smoothly. The hard part was to find a date. Professors are notoriously short on time, so it took one Doodle poll to find a day and another poll to find a time-of-day. But eventually I got my time slot.

Read more…

In September 2017 I have finished my Master's degree in Physics. I was offered a PhD position by my supervisor and gratefully accepted the opportunity. During the master thesis I started to work with lattice simulations, supercomputer programming and was getting into it. Although I already did not want to persue a full career in science, I still wanted to do a bit more research in physics before leaving for the industry.

The thing is that in Germany usually only 25 % of the employees in institutes have permanent positions. And these are usually people who do administration part time. These people are full time professors, administrators of machine shops, computer clusters or something else. The majority of positions are temporary contracts. The rationale seems to be that research benefits from the exchange of ideas, and if people move around the instituions, knowledge is spread. This completely ignores the fact that these are people, eventually wanting to start a familiy and the like. One usually does not get a consecutive contract at the same institution and have to move, often somewhere within the EU. The real chance for a permanent position would be to have it mixed with something permanent, we have IT administrators that are part-time administrators and part-time researchers/teachers. But then it is not really a career in research, it is merely something in academia. All these things set my long-term route, but I did not want to leave research at that moment. So I took the opportunity to do research for another three years.

At the beginning of a PhD the topic is not clear cut. My advisor had a few ideas, and I mostly started working with my valued PhD student coworker Markus to work on his project. There I helped to refactor a C++ code which did tensor contractions. Over the time I learned more of the code, had more and more ideas on improving it. Together we worked on it a lot, it was a really great time. Also I helped to improve the analysis that he needed for his data. Over time it became our analysis, I wrote most of it in the early days. He explained some of the mathematical theory behind it, I implemented a bunch of statistical transformations. On some days we sat there until very late to make some plots really readable, pretty and informative. As the project came to a conclusion, he finished up his dissertation and eventually handed it in. I was super happy for him to finish, and also sad to see him go and move to a different city.

Read more…