Joe Armstrong (programmer)

Joseph Leslie Armstrong (27 December 1950 – 20 April 2019) was a computer scientist working in the area of fault-tolerant distributed systems.

Quotes

The forgotten advantage of concurrent programming

The forgotten advantage of concurrent programming, a full transcription is also available on Erlang Solution website. Recorded in 2019

• I wanted to build fault tolerant systems and pretty soon I realized that you can’t make a fault tolerant system on a computer, because I think in the entire computer might crash, so I needed lots of independent computers.

• "Messages take time", and they propagate through space, there’s no guarantee it gets there.

• I just want to model what’s going on in the real world

• We need to build a world where there are parallel processes communicating through message passing and I thought they cannot have shared memory because if they have shared memory and the remote going to be the crash.

• Why are people just moving the data and not the programs. We could move both of them to some intermediate point in the middle to perform the computation there.

• This is a thing that really scares me, are people developing large applications that they don’t understand.

• Many programs don’t have well-defined interface. They should have.

• I think we seem to have forgotten that things can be small. This way of decomposing systems into small things that I can reason about.

• People keep telling me this, "you’ve got to upgrade your operating system". Then they say, "Well, that’s because of security things." I don’t really have much confidence in them. If they said we have to change it once every 20 years, I could believe that it was reliable, but telling me that I have to change it once every six weeks is crazy.

• One of the things we’ve forgotten, is the importance of protocols and not describing them accurately.

• If you imagine the combination of petabytes memories with LiFi and communication at tens of gigabits per second but the combination and like 10,000 Cray-Ones and a little thing like your fingernail everywhere in every single light bulb that’s like an atomic bomb hitting software. What we’re going to do with it, nobody’s got a clue.

• deployment is a problem because even if somebody made an open source privacy application, it needs 50 million users to take off. Apple and Google and everybody have dominated this way of deploying something to hundreds of millions of people [...] the first one to get a hundred million users wins basically.

• Google knows everything about us but we know nothing about Google.

• I always imagine a historian in 2 or 300 years' time writing the history of this period. It would just be like the Dark Ages, the ages of confusion. Will it end with computer failures that kill millions of people or will it transition into something that is for the benefit of mankind? We don’t know at the moment and I don’t know how long it will be before we know.

The Forgotten Ideas in Computer Science

Faults, The Forgotten Ideas in Computer Science, March 16, 2018

• I had a plan. Learn Emacs. Learn UNIX. And learn programming language. What happened? I didn't learn Emacs. I didn't learn UNIX. And I invented a new programming language.

• Have we got better at programming things in the last 25 years? I think the answer would be no. They've been small improvements.

• I started programming in like 76. There was not enough stuff. There was Fortran and Cobol... And Fortran. Now there's 250, 500, 800 programming languages, one new each week.

• Do we really need to hijack our attention systems every 10 seconds with a banner?

• The methodology for what I was going to do: ask some questions, get some replies, organize result, choose the best things to do.

• How to make a list? Collecting the data, that's the easy bit. Sorting it into categories is slightly more difficult. Shortening the list is the most difficult part. That a really difficult problem.

• If somebody puts a big list of things up, my question is always: what is the most important thing on that list.

• I can only implement one thing at a time. I'm a one thing at a time person. So, if you give me that huge list of things, I will implement the most important thing. Which is it?

• We can't do a list of 20 things. You can't do 20 things in parallel. Do one of them. Do the next.

• Your first question must always be: what is the most difficult part of this problem that I am trying to solve and then try and solve that. If you can't solve the most difficult bit, give up! Seek advice, find somebody who can solve it!

• It's going to fail your project if you take the easy bits first, deliberately. Because you can't solve the difficult bit, your project will fail late than failing early and your company will lose lots of money.

• Four good tools to learn: Emacs, Bash, Make and Shell. You could use Vi, I am not religious here. Make is pretty damn good! I use Make for everything, that good!

• Better algorithms probably gave us a factor of 6 in performance. Changing the programming language going to Prolog to C, that gave us a factor of 15. All the rest comes from hardware. So, wait 10 years, you've got a thousand. wait 20, you get a million in hardware performance.

• If you try to fiddle around with your code to make it efficient, that's the wrong thing to do. Keep on track with the latest hardware. That's the right thing to do.

• Four languages to learn: C, Prolog, Erlang, Javascript.

• Let me just say something about pipes. The great UNIX philosophy was: the output of my program should be the input to your program, to compile things together.

• These [pipes] are killed by GUIs. So you don't have a text flow into a GUI. So, that are GUIs, you can't pipe it together in sequences. That's killed the notion of a pipe and the notion of reusing things.

• We can use Machine Learning to diagnose diseases and we don't need to use machine learning to target advertisements to people. We can use it to cure their cancers. Just think about that.

• I was crap at writing. I learned to ride. I wrote a few books and after, you written a few books, you get good at it. But I was really rubbish at writing to start.

• PHP, brilliant language! Apart from the syntax and the semantics. Because it takes a view that will perform everything in one place.

• Helping your non-technical neighbor. Tell them it's not your fault. Tell them it's crap software and I don't understand this crap.

• Things can be small. They don't have to be gigabytes.

• The web is totally broken. It's not symmetric.Easy to read stuff, very difficult to write stuff. We ave a community of users who engage passively by reading stuff, they do not write stuff.

• Let's unbreak the web. The web is broken. Let's go out and un-break it and let's bring computation back to these supercomputers that we've got in our pockets [...] Let's ensure that our personal data is owned by us and not by large corporations [...] Let's make computing easy again like it was in the past. Let's build apps that could communicate with each other.

Joe Armstrong interviews Alan Kay

Joe Armstrong interviews Alan Kay, Code Mesh London November 3-4, 2016

• I was using Smalltalk and I never actually go through this boundary of sort of simulating the class structure in my head so that I can program itself. It was painful, I used to chose the changelog to find things, I regret you did it.

• I got this Prolog system, and I thought, let's do a message passing object model there. There were an influence with CSP.

• If you do project that guaranteed to succeed, think it could be horribly conservative things. So you can't just do things. You know tat hundred are just fun stuffs and they gonna fails!

• Every Erlang process in the universe should be addressable, and trace backed if we wish they will talk together.

• One of the things which worries me is the silly idea with the cloud. If we just store everything in the cloud, the unintentional side effect of that is we will probably lose all our history. And I am kind of worry to life. two hundred years time and look back and say just sorry, we just lost all the history of the last twenty centuries.

Faults, Scaling and Erlang concurrency

Faults, Scaling and Erlang concurrency, September 24, 2014

• In order to handle failure, you need two machines. [...] You have to be interested in distributed programming. You have to be interesting in parallel computing. You have to be interested in concurrent programming.

• One way I think Erlang was a kind of software emulating Tandem machine.

• If you managed to scale horizontally, you can scale.

• You can both kept fault tolerance and scalability. You can have both or none of them

• Shared memory is evil.

• Defensive programming in C, is only necessary because you have only have a single thread of computation. If you have a sequential language and it crashes, you lost everything.

• In Erlang, you have as many processes you want. You can arrange the processes observing each other. If you have got half million processes to do something, what is it matter if few thousand of them fail?

• The cornerstones of fault tolerant programming are to isolate errors to make sure if one thing crash it does not crash anything else. That's what the processes in operating system do.

• Bad concurrency model, I think, is something that make a lot of programming artificially difficult.

• If you can't detect failures, you are unable to recover from them. That's an obvious principle you need.

• We need to know why things fails. If we want to improve our system, we need to put that information into that system into logs or something.

• Processes are isolated by design. Context switching are very lightweight. The processes by design cannot damage each other.

• Our goal is actually to have about 75% utilization if we had one hundred CPU we would hope that our program will go 75% faster without tweaking it.

Modeling the World with Processes Objects Functions of Relations

26 Years with Erlang, September 22, 2014

• Composing computation. Why do we do compose computation? Because We want to make reusable things out of small things. We take small things to make bigger things.

• In Haskell, C or Java, or any these languages, that function composition will fail at compile time. Type system will say: we'll not allow you to compose things together because the types are wrong. But in Erlang, you can do anything you bloody would feel like just found together and know it at run time not at compile time. Some people think that is not so good. Some people think that is pretty cool. I will tell you why it's pretty cool. It's actually late binding, it makes decision late.

• Hidden state in object so in any object programming language means you can't compose things. Hidden state is a just state effect stopping you to composing things.

• In pure functional language, you are not allowed to tinker around with the state, So the state go into your function, something should happen and the state comes out as well. In pure functional language, you got in input and the state, and what's come out is the output and the new state.

• Object Oriented Programming, this is the art of hiding side effects. If you can do it successfully you can write this in program. If you do it unsuccessfully, you get in a sucking mess.

• The different ways of hiding the state are the things that different functional programming languages offer. In Haskell they call it monads, in Erlang they call it processes.

• Pipelines is the most important word. So a Monad is a pipeline. It allows you to pipe an output of a function to the input of another function. They rediscover the pipe.

• We could of course redefine compose to a version of compose that work with debugging with debugging things. That would be horrible. That would mean we would need to compose pure functions. We would need to compose functions for debugging and everything on the planet.

• Instead of handy compose, we'll keep compose exactly the same and will write a little function which takes a function that does not have debugging and return the function that does debugging. And we have a function that accept only one argument, the un-debugging version, we'll add another little function that automatically add an argument to it. When we done that, we can bind call compose on it and everything will work and will be nice. And will be peace on the earth and everybody will have a good time.

• The nice thing about pipeline is that quite easy to see and you can do all the steps in the pipeline in parallel.

• In Erlang, F(G(X)) is used for small steps. Pipes are used for big steps. The big steps semantics is processes. It's messages between processes. Processes are used for big computation. When you got F(G(X)) is a small step, and we map small steps on the same core. We map big steps on processes. Processes map on multi-core. The system just does it for you.

• If you have know idea where the leak is coming from, you'll want call a professional plumber who will have the equipment to locate the pipes in your walls [...] So, he's talking about software!

• Middle men, this is the key abstraction in Erlang. This is the model of the world. Everything in the world is processes. When ever you got any problem what so ever, it does not matter what was of problem is, you put this transducer thing in the middle. It translate the way things talk to each other in the outside world and internally.

• The Middle man is the thing that imposed Erlang only in the Erlang world. If people try to talk to each other, the complexity of the program is O(N²). But, if we put middle men here, this turns the complexity to O(N).

The Mess We're In

The Mess We're In. Recorded in 2014.

• How many machines boot in 60 milliseconds? When did it come wrong?

• I think software are getting worse and worse in time.

• We should get the Congressional Medal for creating employments because we have created billions of man hours of maintenance work for people in the future. We are the job creators of the future.

• Comments are really good, so I would advise you to write comments. Write really big and expensive comments.

• There is another word for really big comments: a book!

• If you don't have a book, you don't know how a bloody program work!

• What's legacy code? That's dead programmers stuff, not only there are no comment in it, you can't ask people because they are dead. It written in languages that nobody understand, there is no specification and yet it works beautifully!

• Sometimes changing one line of legacy code is incredibly difficult than totally rewriting the entire stuff.

• The number of states of three variables in javascript could have is greater of the total number of the atoms of the planet.

• What did it work for this guy and not for me? Well, it's because machines were in different states when we performed those operations. The number of possible state of the machine is a really great number.

• Computer systems fail. We have to deal with it, we can't ignore it.

• We can't handle failures on one computer.

• That's quite easy to make things that are not fault tolerant and not scalable.

• People says notation does not matter, they don't like Erlang notation [...] but notation does matter.

• Without Google and Stackoverflow programming would be impossible.

• A lot of systems actually break the laws of physics.

• Two phases commit breaks laws of physics.

• You should not write systems that violate laws of physics.

• Files and systems they mutate all the time.

• If we combine git and bittorrent we got gittorrent.

• Computing was about controlling complexity and we have failed miserably.

• We need mechanism to make things smaller and smaller.

• Computers are becoming a big environmental threat. They are using more energy than air traffic.

26 Years with Erlang

26 Years with Erlang, July 13, 2013

• I thing the first company that's a pure Erlang worth a billion dollars.

• I was programming Prolog []... What happens when a Prolog program goes wrong? it says no. There "no" is not a very good error. Core damage is much more informative for no.

• One thing I wanted to do with Prolog, pick some sort of better things. Better stuff should happen when it fails. The other problem with prolog is that you could only do one thing, and you can only have one process. So I wanted and I thought Prolog was really great. I made an algebra to describe telephony in Prolog but the only trouble was you could only describe one process and if they didn't do what you'd expected them to do when you got failure, it didn't work.

• The spec does not tell me what to do. What do I program? [...] I thought this is crazy, we are building products with specifications and things, and the specifications doesn't say what's supposed to happen. Then when you get to this point in the code where you don't know what's going to happen, everybody knows how programs it. I thought that's crazy because I didn't know and I thought everybody could interpret that differently and they could make an ad-hoc decision. So I thought to myself, the only sensible thing to do is crash. We just crash this call because we don't know what to do.

• In a file system, if a program opens a file and does something and then crashes, the operating system closes the file for you. It's not still open and then you can use it again. So I thought to myself, well, a call is like a file. So, what happens if you do this hook, you crash the program. Somebody else will watch (the operating system) will watch) and it will close the call just like closing a file. That was the origin of this thing that's now I call Let It Crash.

• This was actually fast to build the prototypes of the telephony system but they were running very slowly. at that stage we said: let's make a product out of this. We thought the semantics were okay but the performance wasn't.

• I think things like PowerPoint have sort of destroyed creativity.

• I will never ever make any statement about performance that is not based on measurement and I will never ever advise anybody on how to write their code without saying write it and measure. I just will not speculate. I think people who do speculate about this should not speculate about it and they should write it, run it and measure it.

• I am the world's worst C programmer because it's so complicated I can't understand it [...] All I know is that whenever I write C, my pointers are always missed by one from where they're supposed to go.

• You're a programmer, you've just written 100 lines of codes that C++ programmers have failed to do. You want to show it to people because you're quite pleased. You think they're going to be pleased...

Fault Tolerance 101

Fault Tolerance 101. 12 July 2013.

• You can't do anything unless you detect the fault.

• You don't want your fault creep into the system and damage other parts of the system. you need to build firewall between things to stop them from propagating.

• If the entire computer crashes, you're screwed. you can't really do fault tolerant computation from one machine.

• you can basically make things as reliable as you by having lot of machines.

• Don't be deluded into thinking that is sort of ultimately fault tolerant. You can only provision it for the faulty things that might happen and then for a large class of things it could happen.

• If you do something and it fails you can do it again, it will work the second time. That's because quite often you are using something underneath where you've missed a message or messages are coming in wrong order.

• The problem is more than just the should do when it works, it should also say what it should do in presence of failures that you could reasonably predict.

• You have to analyze the consequences of a failure and that to the specification.

• Erlang was specifically designed for fault tolerance, it was not designed for anything else.

Over a Century of programming

Over a Century of Programming, June 2013

• This [showing his smartphone] is about a million time more powerful than the computer but the computer was developed on.

• We wanted to build big distributed systems, fault tolerant systems. The sort of systems that everybody wants to build today. It's not just about telecom. At the time it was just about telecom but now, it's not just about telecom. Virtually, every big systems you want to build has to do these things. You want to connect millions of people and you don't want if the system fail in anyway, you don't want these individual transaction to be broken. We started to working on that, way back. That sort of way we got Erlang.

• A technology that nobody use, just dies.

Sherlock's Last Case

Sherlock's Last Case. 12 July 2013.

• This is a letter about a conference. Does it go in letter or in conferences? This is a categorization problem, as soon you put categories into things you don't know what category to use. This is the problem in object oriented programming, this is why object oriented programming is bloody stupid.

• Object oriented programming asked you to put things into categories, into classes. Concurrency oriented programming asks you to identify the concurrency and the problem. The concurrence in the problem is a property of the real world. If you are doing real world modeling you can see it, you can smell it, you can eat it

• I have a problem with categorization problem because I cannot say this thing got to be in that module or that class.

• I don't like the fact programmers are isolated. I hate all these git rubbish stuff. I mean, this is horrible because it makes you think that programmers are individuals and not part of a community. It makes them thinks that what they are doing is isolated.

• When I write code I always think somebody else must have written this before me and I can't find it. Why do we write code if we can't find it with Google. If we can't really find it, then we can write the code. Lot of the code is pain in the arse, it's just boring stuff.

• People are inventing new programming language all the time but I think that's not the big problem is actually writing programs. I think the big technical problem in building system is what I call "the connecting things together problem". It's a version nightmare problem. It's given things in different languages and actually connecting them together.

• If I want to make a programming language, I would not make a programming language, I would make a protocol description language where you can formally describe protocols and you can formally validate the components are obeying the protocol. I put these between everything that communicates in a form of a contract and I would not like to serve contract violations.

• I do not think we can build systems that are large with large components.

• I don't think a program should be bigger than possibly 128 kilobytes, possibly 64 kilobytes. It should be possible to understand that program in its entirety.

• I think we have to think about how we can make systems out of very small components with very strictly defined protocols between them.

• I think we need to start describing configuration languages that describe how the thing is built and configured.

• To me, programs are black boxes. You send a message in and you get a message out. I don't care if it's written in Scala or Haskell or Erlang or Python or Ruby. I care about this relationship between inputs and outputs.

• API in programming languages is totally stupid because they do not capture order.

• The API we use in programming languages are not adequate for describing the order in which we can perform the API calls. They are basically useless.

• You know, there are two types of languages, there are languages that people bitch about and there are languages that nobody uses.

• The terabyte boundary is going to cause movies to have the same status as films and digital books.

• That's enormously interesting, the petabyte change because what's gonna happen is all human knowledge, every book that ever been written will fit into 138 terabytes of memory. That will mean the petabyte of memory can store all human knowledge and we can give it to every person on the planet. If we do that it's totally going to change society.

• [in the next 5, 10 or 15 years] I think we will see repressive states to try and stop technology. We will find free states try and promote it.

• I find it totally absurd that the word intellectual and property, those two things can be put into the same sentence and you connect intellectual property with property. As if an idea can be owned and if you can get money for it. It's not physical! Physical things, you can own! You can't own ideas!

• The other thing you have got to do is take the data out of the cloud and put it into your home cloud. Make your own home cloud.

• We are computer scientists, we are slaves who works for these big companies [Google, Facebook].

The How and Why of Fitting Things Together

The How and Why of Fitting Things Together, Erlang User Conference 2013

• Why do the designers to this programming language want correctness? It's because, if you write program in erlang, and it turns to be incorrect, I got the blame. But if it works you got the credit.

• Erlang is like Meccano. Meccano is very good. You build wonderful things with meccano. C++ is like Lego. Lego is wonderful. But there is any glue. The glue that you want to use to glue Lego and Meccano just does not work.

• This is fantastic, this a quantum leap in abstraction. Most people are being looking inside the black boxes and trying to understand how they work but I think they are looking at the wrong place.They should have been looking inside the black box to understand programming works and looks on the connected things to understand what do they do.

• Everything is interesting, everything does connect, but anything don't work.

• why do we write things from scratch all the time and reimplementing things? because it's quicker than finding another programming language that does it. We can't find this program, and if we find it, it does something slightly different to the one we wanted and it's quicker to rewrite a new program than modify the old program.

• Joe's law: Frameworks grow in complexity until nobody can use them.

• Messages are like files. We don't care how it was created.

• Pipes are wonderful. They are doing wonderful things.

• Bad ideas in computer science and anywhere are sticky. If the first idea was a bad idea, then because it works people will sort of repeat it.

• The middle man brings conceptual integrity to the system.

A Few Improvement to Erlang

A Few Improvement to Erlang, Erlang User Conference 2012

• I could not understand how people could have get started with PHP because there was not message at all.

• I thought if you multiply two integers together, you get an integer. But apparently you don't in PHP. You get a floating point number, which gets bigger and bigger and that's okay. Until you get to factorial(171), and then you get "INF". I don't know what "INF" is. I think it's infinity actually. But in that case, it's probably wrong, because infinity is awfully big. Bigger than factorial(171).

• So what's wrong with the shell? and what's wrong with Erlang? It's not LISP!

• Meta-Programming Erlang is not really easy because of the way the trees are parsed.

• I don't like modules. It's a sort of love/hate relationship. The problem with modules is the classification problem. When you have written a function you don't know in what modules you put it in.

• Modules change with time, and there is no sense of time in programming development. If you look at something like git, the way you deal with time is checksum, timestamp...

• When we write code, we forgot about the bigger picture.

• Finally after hours of googling, I can write my program. That what's I call research. The research takes 3 hours, writing the program takes 2 minutes, for the final version. You publish the program, and you throw away the research. That's not smart. Somebody later cannot figure out how you did it because all he's see is the end result.

• The nice thing about the Erlang documentation is everything is done in XML. XML is absolutely brilliant for marking up text.

• Documents are full of paragraphs. But the paragraphs don't have name. So, we can't talk about the paragraphs.

Programming Erlang - Software for a Concurrent World

Programming Erlang, Software for a Concurrent World, The Pragmatic Programmers, 2007

• When I'm writing a program, my approach is to "write a bit" and then "test a bit." I start with a small module with few functions, and then I compile it and test it with a few commands in the shell. Once I'm happy with it, I write a few more functions, compile them, test them, and so on. (page 60)

Making Reliable Distributed Systems in the Presence of Software Errors

Making Reliable Distributed Systems in the Presence of Software Errors, December 2003

• The inability to isolate software components from each other is the main reason why many popular programming languages cannot be used for making robust system software. (page 32)

Erlang The Movie

Erlang the Movie, 1990

• The clarity of programming languages have several advantages over traditional languages. For example, programs in such languages are considerably shorter than the equivalent programs in imperative languages. They also encourage clear programming styles.

• Up to now, declarative programming languages such as Prolog or ML have not been used for real-time applications.

• So we have had to add a notion of concurrency in real-time to our language. We also added a sophisticated error detection mechanism. This allows us to program robust real-time systems.