Docker

in this video I will teach you all the main concepts of Docker including getting your first hands-on experience with it so if you have to use Docker at work or if you need to learn Docker to level up your engineering skills and need to get started fast and understand all the main Concepts and learn basics of how to work with Docker this crash course is exactly right for you first we'll start by explaining what Docker is why was it even created basically what problems it solves in engineering and how it helps in software development and deployment process so you will understand exactly why Docker is such a big deal and why it has become so popular and widely used in IT projects and as part of a virtualization solution Docker being an improvement over virtual machines or the next Evolution step I will also explain the difference between virtual machine and Docker and what are the advantages of Docker in this comparison after we've understood why we want to use Docker in the first place we will install Docker and learn how to actually work with it we will learn the concepts of Docker images containers Docker registry public and private Registries and we will run containers locally based on some of the images available on Dockers public registry called Docker Hub we will also learn the concept of creating your own images and learning about a Docker image blueprint called Docker file and of course we will see all these in action and learn all the docker commands for pulling images running containers building your own Docker image Etc we will also learn about versioning images with image text and finally after you've learned how to work with Docker I will also explain with graphical animations how Docker fits in the big picture of software development and deployment process so by the end of this video you will feel way more confident about your knowledge and understanding in Docker and and can easily build on that Foundation knowledge to become a Docker power user if you want to and under the video description I will provide some resources to learn even more about Docker and become more advanced in it but before we jump right in it seems like many of you watching the videos on our channel are still not subscribed so if you're getting some value out of the free tutorials I put out regularly on this channel be sure to subscribe not to miss any future videos or tutorials I would also be happy to connect with you on my other social media accounts where I post behind the scenes content weekly updates and so on so hope to connect to you there as well well I'm super excited to teach you all these so let's get into it let's start with the most important question what is Docker why was it even created and what problem does it solve in simple words Docker is a virtualization software that makes developing and deploying applications very easy much easier compared to how it was done before Docker was introduced and Docker does that by packaging an application into something called a container that has everything the application needs to run like the application code itself its libraries and dependencies but also the runtime and environment configuration so application and its running environment are both packaged in a single Docker package which you can easily share and distribute now why is this a big deal and how are applications actually developed and deployed before Docker was introduced let's see that to understand the benefits of Docker more clearly so how did we develop applications before containers usually when you have a team of developers working on some application they would have to install all the services that application depends on or needs like database Services Etc directly on their operating system right for example if you're developing a JavaScript application and you need a postgresql database maybe you need a redis for caching mosquito for messaging like you have a microservices application now you need all these Services locally on your development environment so you can actually develop and test the application right and every developer in the team would then have to go and install all those Services configure and run them on their local development environment and depending on which operating system they're using the installation process will be different because installing postgresql database on Mac OS is different from installing it on a Windows machine for example another thing with installing Services directly on an operating system following some installation guide is that you usually have multiple steps of installation and then configuration of the service so with multiple commands that you have to execute to install configure and set up the service the chances of something going wrong and error happening is actually pretty high and this approach or this process of setting up a development environment for a developer can actually be pretty tedious depending on how complex your application is for example if you have 10 services that your application is using then you would have to do that installation 10 times for each service and again it will differ within the team based on what operating system each developer is using now let's see how containers solve some of these problems with containers you actually do not have to install any of the services directly on your operating system because with Docker you have that service packaged in one isolated environment so you have postgresql with a specific version packaged with its whole configuration inside of a container so as a developer you don't have to go and look for some binaries to download and install on your machine but rather you just go ahead and start that service as a Docker container using a single Docker command which fetches the container package from internet and starts it on your computer and the docker command will be the same regardless of which operating system you're on and it will also be the same regardless of which service you are installing so if you have 10 services that your JavaScript application depends on you would just have to run 10 Docker commands for each container and that will be it so as you see Docker standardizes the process of running any service on your development environment and makes the whole process much easier so you can basically focus and work more on development instead of trying to install and configure services on your machine and this obviously makes setting up your local development environment much faster and easier than the option without containers plus with the docker you can even have different versions of the same application running on your local environment without having any conflict which is very difficult to do if you are installing that same application with different versions directly on your operating system and we will actually see all of this in action in the demo part of this video now let's see how containers can improve the application deployment process before containers a traditional deployment process would look like this development team would produce an application artifact or a package together with a set of instructions of how to actually install and configure that application package on the server so you would have something like a jar file for Java application or something similar depending on the programming language used and in addition of course you would have some kind of database service or some other services that your application needed also with a set of instructions of how to configure and set it up on the server so that application could connect to it and use it so development team would give that application artifact or package over to the operations team and the operations team would handle installing and configuring the application and all its dependent services like database for example now the problem with this kind of approach is that first of all you need to configure everything and install everything again indirectly on the operating system which I I mentioned in the development context that is actually very error prone and you can have various different problems during the setup process you can also have conflicts with dependency versions where two services are depending on the same library for example but with different versions and when that happens it's going to make the setup process way more difficult and complex so basically a lot of things that can go wrong when operations team is installing and setting up application any services on a server another problem that could arise from this kind of process is when there is a miscommunication between the development team and operations team because since everything is in a textual guide like an instruction list of how to configure and run the application or maybe some kind of checklist there could be cases where developers forget to mention some important step about configuration and when that part fails the operations team have to go back to developers and ask for more details and input and this could lead to some back and forth communication until the application is successfully deployed on the server so basically you have this additional communication overhead where developers have to communicate in some kind of textual graphical whatever format how the application should run and as I mentioned this could lead to issues and miscommunications with containers this process is actually simplified because now developers create an application package that doesn't only include the code itself but also all the dependencies and the configuration for the application so instead of having to write that in some textual format and document they basically just package all of that inside the application artifact and since it's already encapsulated in one environment the operations people don't have to configure any of this stuff directly on the server so it makes the whole process way easier and there is less room for issues that I mentioned previously so the only thing now that operations team need to do in this case is to run a Docker command that gets the container package that developers created and runs it on the server the same way operations team will run any services that application needs also as Docker containers and that makes the deployment process way easier on the operation side now of course the operations team will have to install all and set up the docker runtime on the server before they will be able to run containers but that's just one-time effort for one service or one technology and once you have Docker runtime installed you can simply run Docker containers on that server now at the beginning I mentioned that Docker is a virtualization tool just like a virtual machine and virtual machines have been around for a long time so why did Docker become so widely adopted what advantage is it has over virtual machines and what is the difference between the two for that we need to see a little bit of how Docker works on a technical level I also said that with Docker you don't need to install Services directly on operating system but in that case how does Docker run its containers on an operating system now in order to understand all this let's first look at how an operating system is made up operating systems have two main layers you have the operating system kernel and the operating system Apple locations layer and kernel is the part that communicates with the hardware components like CPU memory storage Etc so when you have a physical machine with all these resources and you install operating system on that physical machine the kernel of the operating system will actually be the one talking to the hardware components to allocate resources like CPU memory storage Etc to the applications then running on that operating system and those applications are part of the applications layer and they run on top of the kernel layer so kernel is kind of a middleman between the applications that you see when you interact with your computer and the underlying Hardware of your computer and now since Docker and virtual machine are both virtualization tools the question is what part of the operating system they actually virtualize and that's where the main difference between Docker and virtual machines actually lie so Docker virtualizes the applications layer this means when you run a Docker container it actually contains the applications layer of the operating system and some other applications installed on top of that application layer this could be a Java runtime or python or whatever and it uses the kernel of the host because it doesn't have its own kernel the virtual machine on the other hand has the applications layer and its own kernel so it virtualizes the complete operating system which means that when you download a virtual machine image on your host it doesn't use the host kernel it actually puts up its own so what is this difference between Docker and virtual machine actually mean first of all the size of the docker packages or images are much smaller because they just have to implement one layer of the operating system so Docker images are usually a couple of megabytes large virtual machine images on the other hand can be a couple of gigabytes this means when working with Docker you actually save a lot of disk space you can run and start Docker containers much faster than virtual machines because virtual machine has to put up a kernel every time it starts while Docker container just reuses the host kernel and you just start the application layer on top of it so while virtual machine needs a couple of minutes to start up Docker containers usually start up in a few milliseconds the third difference is compatibility so you can run virtual image of any operating system on any other operating system host so on a Windows machine you can run a Linux virtual machine for example but you can't do that with Docker at least not directly so what is the problem here let's say you have a Windows operating system with Windows kernel and its application layer and you want to run a Linux based Docker image directly on that Windows host the problem here is that Linux based Docker image cannot use the windows kernel it wouldn't need a Linux kernel to run because you can run a Linux application layer on a Windows kernel so that's kind of an issue with Docker however when you're developing on Windows or Mac OS you want to run various Services because most containers for the popular services are actually Linux based also interesting to note that Docker was originally written and built for Linux but later Docker actually made an update and developed what's called Docker desktop for Windows and Mac which made it possible to run Linux based containers on Windows and Mac computers as well so the way it works is that Docker desktop uses a hypervisor layer with a lightweight Linux Distribution on top of it to provide the needed Linux kernel and this way make running Linux based containers possible on Windows and Mac operating systems and by the way if you want to understand more about virtualization and how virtual machines work and what a hypervisor for example is you can watch my other video where I explain all of that in detail so this means for local development as an engineer you would install Docker desktop on your Windows or Mac OS computer to run Linux based images which as I mentioned most of the popular Services databases Etc are mostly Linux based so you would need that and that brings us to the installation of Docker in order to do some demos and learn Docker in practice you would first need to install it so in order to install Docker you just go to their official page for installation guide and follow the steps because Docker gets updated all the time the installation changes so instead of me just giving you some comments that may work now but we'll get updated in the future you should always refer to the latest documentation for installation guide for any tool so if we search for Docker desktop installation click on one of those links like install on windows so that's the docker desktop the tool that I mentioned that solves this problem of running Linux based images on a different operating system but it actually includes a lot of other things when you install it so what are you exactly installing with Docker desktop and you see exactly what's included in there so basically get the docker service itself it's called Docker engine that's the main part of the docker that makes this virtualization possible but when we have a service we need to communicate with that right so we need a client that can talk to that service so Docker desktop actually comes with a command line interface client which means we can execute Docker commands on a command line to start containers to create containers start stop them remove them Etc and do all kinds of things and it also comes with a graphical user interface client so if you're not comfortable working with command line you can actually use the graphical using interface where you can do all these things but in a nice user-friendly UI so you get all these things when you install Docker desktop basically everything that you need to get started with Docker and of course depending on which operating system you're on you're going to choose that one Mac windows on Linux so let's click on one of those and you basically just follow the instructions you have some system requirements you have to check things like the the version of your Mac OS how much resources you're going to need and you also have the options for Mac with Intel or Mac with apple silicon so you can toggle between those and basically just choose the guide that matches your computer specifications and once you have that check the system requirements go ahead and click on one of those in my case I have mac with Intel chip so I would click on this one and that's actually the docker desktop installer so if I click it's going to download this DMG image and once it's downloaded you basically just follow the steps described here right you double click on it open the application and so on on on and same for Windows if your windows you basically click on this one and download Docker desktop for Windows and make sure to check the system requirements and kind of prepare everything you need for starting Docker generally for latest versions of Windows Mac or whatever operating system it should be pretty easy and straightforward to install Docker so go ahead and do that once you're done with installation you can simply start the service by searching Docker and if I click on it you will see right here that it's actually starting up Docker service for Docker engine and there you go it's running and this view here that you're seeing this window is actually the graphical user interface of Docker that I mentioned so that's the client that you can use to interact with the docker engine so you have a list of containers running currently so there's no list same with images if I switched images I have cleaned up my environment so I'm starting with scratch with empty State just like you so we're ready to start using Docker But first you may be wondering what are images and that's what I'm gonna explain next because it's a very important Concept in docker now mentioned that Docker allows to package the application with its environment configuration in this package that you can share and distribute easily so just like an application artifact file like when we create a zip or tar file or a jar file which you can upload to a artifact storage and then download on the server or locally whenever you need it and then package or artifact that we produce with Docker is called a Docker image so it's basically an application artifact but different from jar file or from other application artifacts it not only has the compiled application code inside but additionally has information about the environment configuration it has the operating system application layer as I mentioned plus the tools like node npm or Java runtime installed on that depending on what programming language your application was written in for example you have a JavaScript application you would need node.js and npm to run your application right so in the docker image you would actually have node and npm installed already you can also add environment variables that your application needs for example you can create directories you can create files or any other environment configuration whatever you need around your application so all of the information is packaged in the docker image together with the application code and that's the great advantage of Docker that we talked about and as I said the package is called an image so if that's an image what is a container then well we need to start that application package somewhere right so when we take that package or image and download it to server or your local computer laptop we want to run it on that computer the application has to actually run and when we run that image on an operating system and the application inside starts in the pre-configured environment that gives us a container so a running instance of an image is a container so a container is basically a running instance of an image and from the same image from one image you can run multiple containers which is a legitimate use case if you need to run multiple instances of this same application for increased performance for example and that's exactly what we were seeing here so we have the images these are the application packages basically and then from those images we can start containers which we will see listed right here which are running instances of those images and I also said that in addition to the graphical user interface we get a command line interface client Docker client that can talk to Docker engine and since we installed Docker desktop we should have that Docker CLI also available locally which means if you open your terminal you should be able to execute Docker commits and Doc recommends we can do anything for example we can check what images we have available locally so if I do Docker images that will give me a list of images that I have locally which in this case I don't have any which we saw in the graphical user interface and I can also check the containers using a command docker occur PS PS PS and again I don't have any running containers yet now before moving on I want to give a shout out to Ned Hopper net Hopper's Cloud platform called kubernetes application operations offers an easy way for devops teams to deliver manage upgrade connect secure and monitor applications in one or more kubernetes clusters with this platform they basically create this virtual Network layer that connects multiple environments for example if you have multiple Cloud platforms and multiple kubernetes clusters even your own on-premise data center where your application gets deployed you can connect all these in one virtual Network so you can deploy and operate your kubernetes workloads as if it was in one cluster or one infrastructure environment and the GitHub Centric approach they use offers the visibility to know who did what and when for both your infrastructure and application so with net Hopper Enterprises can automate their operations and instead of building an own platform devops teams can focus on what matters the most which is releasing more application features faster so check them out you can actually sign up for a free account and take it for a spin to see if net Hopper is the right solution for you now it's clear that we get containers by running images but how do we get images to run containers from let's say we want to run a database container or redis or some log collector service container how do we get their Docker images well that's where Docker Registries come in so there are ready Docker images available online in image storage or registry so basically this is a storage specifically for Docker image type of artifacts and usually the company is developing those services like redis mongodb Etc as well as Docker Community itself will create what's called official images so you know this mongodb image was actually created by mongodb itself or the docker community so you know it's an official verified image from Docker itself and Docker itself offers the biggest Docker registry called Docker Hub where you can find any of these official images and many other images that different companies or individual developers have created and uploaded there so if we search for Docker hub right here you see Docker Hub container image Library and that's how it looks like and you don't actually have to register or sign up on Docker Hub to find those official images so anyone can go on this website and basically browse the container images and here in search bar you can type any service that you're looking for for example redis that I mentioned and if I hit enter you will basically see a list of various radius related images as well as the ready service itself as a Docker image and here you have this batch or label that says Docker official image for example for the reddish image that we are going to choose here you see that it is actually maintained by Docker Community the way it works is that Docker has a dedicated team that is responsible for reviewing and Publishing all content in the docker official images and this team works in the collaboration with the technology creators or maintainers as well as security expert words to create and manage those official Docker images so this way it is ensured that not only the technology creators are involved in the official image creation but also all the docker security best practices and production best practices are also considered in the image creation and that's basically the description page with all the information about how to use this Docker image what it includes Etc Etc Etc and again as I said Docker Hub is the largest Docker image registry so you can find images for any service that you want to use on Docker Hub now of course technology changes and there are updates to Applications those Technologies so you have a new version of redis or mongodb and in that case a new Docker image will be created so images are versioned as well and these are called image tags and on the page of each image you actually have the list of versions or tags of that image listed right here so this is for redis and if I search for postgres for example foreign you will see different image tags for postgres image also listed here so when you're using a technology and you need a specific version you can choose a Docker image that has that version of the technology and there is a special tag that all images have called latest so right here you see this latest tag or here as well in the recent text so latest tag is basically the latest the last image that was built so if you don't specify or choose a version explicitly you basically get the latest image from the docker Hub so now we've seen what images are and where you can get them so now the question is how do we actually get the image from Docker Hub and download it locally on our computer so we can start a container from that image so first we locate the image that we want to run as a container locally for our demo I'm going to use an nginx image so go ahead and search for nginx which is basically a simple web server and it has a UI so we will be able to access our container from the browser to validate the container has started successfully that's why I'm choosing nginx and here you have a bunch of image tags that you can choose from so the second step after locating the image is to pick a specific image tag and note that selecting a specific version of image is the best practice in most cases and let's say we choose version 1.23 so we're choosing this tag right here and to download an image we go back to our terminal and we execute docker pull comment and we specify the name of the image which is nginx so you have that whole command here as well so that's basically the name of the image that you have written here so that's nginx and then we specify the image tag by separating it with a column and then the version 1.23 that's what we chose that's the whole command so Docker client will contact Docker Hub and it will say I want to grab the nginx image with this specific tag and download it locally so let's execute and here we see that it's pulling the image from the image registry Docker Hub and the reason why we don't have to tell Docker to find that image on Docker Hub is because Docker Hub is actually the default location where Docker will look for any images that we specify right here so it's automatically configured as a location for downloading the images from and the download happened and now if we execute Docker images command again as we did here we should actually see one image now locally which is nginx with an image tag 1.23 and some other information like the size of the image which is usually in megabytes as I mentioned so we have an image now locally and if we pull an image without any specific tag so we do this basically Docker pull name of the image if I execute this you see that it is pulling the latest image automatically and now if I do Docker images again we're going to see two images of nginx with two different texts right so these are actually two separate images with different versions cool now we have images locally but obviously they're only useful when we run them in a container environment how can we do that also super easy we pick the image we already have available locally with the tag so let's say we want to run this image as a container and we execute Docker run command and with the name of the image and the tag super easy and let's execute and that command actually starts the container based on the image and we know the container started because we see the logs of nginx service starting up inside the container so these are actually container logs that we see in the console so it's launching a couple of scripts and right here we have start worker processes and the container is running so now if I open a new terminal session like this and to Docker PS I should actually see one container this one here in the running container list and we have some information about the container we have the ID we have the image that the container is based on including the tag when it was created and also the name of the container so we have the ID and name of the container this is the name which Docker actually automatically generates and assigns to a container when it's created so it's a random generated name now if I go back here you see that these locks the container logs actually are blocking the terminal so if I want to get the terminal back and do Ctrl C exit the container exits and the process actually dies so now if I do Docker PS you will see that there is no container running but we can start a container in the background without it blocking the terminal by adding a flag called minus D which stands for detached so it detaches the docker process from terminal if I execute this you see that it's not blocking the terminal anymore and instead of showing the logs from nginx starting up inside the container it just locks out the full ID of the container so now if I do Docker PS here in the same terminal I should see that container running again and that's basically the ID or the part of the this full ID string shown here but when we start a container in the background in a detached mode you may still want to see the application logs inside the container so you may want to see how did nginx start up what did it log actually so for that you can use another Docker command called Docker locks with the container ID like this and it will print out the application logs from the container now in order to create the container the nginx container we first pull the image and then we created a container from that image but we can actually save ourselves the pull command and execute run command directly even if the image is not available locally so right now we have these two images available locally but in the docker run command you can actually provide any image that exists on Docker Hub it doesn't necessarily have to exist locally on your computer so you don't have to pull that first so if I go back we can actually choose a different image version let's choose 1.22 Dash Alpine so this image tag which we don't have locally or of course this can be completely different service it doesn't matter so basically any image that we don't have locally you can run directly using Docker run command so what it does is first it will try to locate that image locally and if it doesn't find it it will go to Docker Hub by default and pull the image from there automatically which is very convenient so it does both in one command basically so it downloaded the image with this tag and started the container and now if we do Docker PS we should have two containers running with different nginx versions and remember I said Docker solves the problem of running different versions of the same application at once so that's how simple it is to do that with Docker so we can actually quit this container and now again we have that one nginx container with this version now the important question is how do we access this container well we can't right now because the container is running in the closed Docker Network so we can't access it from our local computer browser for example we need to First expose the container to our local network which may sound a little bit difficult but it's super easy so basically we're going to do what's called a port binding the container is running on some Port right and each application has some standard port on which it's running like nginx application always runs on Port 80 radius runs on Port 6379 so these are standard ports for these applications so that's the port where container is running on and for nginx we see the ports under the list of ports here application is running on Port 80 inside the container so now if I try to access nginx container on this port on Port 80 from the browser and let's try to do that we're eating and hit enter you see that nothing is available on this port on localhost so now we can tell Docker hey you know what bind that container Port 80 to our local host on any port that I tell you on some specific Port like 8080 or 9000 it doesn't actually matter so that I can access the container or whatever is running inside the container as if it was running on my Local Host Port 9000 and we do that with an additional flag when creating a Docker container so what we're going to do is first we're going to stop this container and create a new one so we're going to do Docker stop which basically stops this running container and we're going to create a new container so we're going to do Docker run nginx the same version and we're going to find it in the background in detached mode now we're going to do the port binding with an additional flag minus p and it's super easy we're telling Docker the nginx application Port inside container which is 80. please take that and find that on a host localhost on Port whatever 9000 for example right that's the port I'm choosing so this flag here will actually expose the container to our local network or localhost so these nginx process running in container will be accessible for us on Port 9000. so now if I execute this let's see that container is running and in the port section we see a different value so instead of just having 80 we have this port binding information so if you forgot which Port you chose or if you have 10 different containers with Docker PS you can actually see on which Port each container is accessible on your Local Host so this will be the port so now if I go back to the browser and instead of localhost 80 we're going to type in localhost 9000. and hit enter there you go we have the welcome to nginx page so it means we are actually accessing our application and we can see that in the logs as well Docker locks container ID and there you go this is the log uh that nginx application produced that it got a request from MEC or Mac OS machine Chrome browser so we see that our request actually reached the nginx application running inside the container so that's how easy it is to run a service inside container and then access it locally now as I said you can choose whatever Port you want but it's also pretty much a standard to use the same port on your host machine as the container is using so if I was running a MySQL container which started at Port 3306.

I would bind it on localhost 3306. so that's kind of a standard now there's one thing I want to point out here which is that Docker run command actually creates a new container every time it doesn't reuse the container that we created previously which means since we executed Docker run command a couple of times already we should actually have multiple containers on our laptop however if I do Docker PS I only see the running container I don't see the ones that I created but stopped but those containers actually still exist so if I do Docker PS with a Fleck a and execute this gives you actually a list of all containers whether they are running or stopped so this is the active container that is still running and these ones are the stopped ones it even says exited 10 minutes ago six minutes ago whatever so we have four containers with different configuration and previously I showed you Docker stop command which basically stops an actively running container so we can stop this one and now it will show it as a stopped container as well exited one second ago but the same way you can also restart a container that you created before without having to create a new one with Docker run command so for that we have a Docker start and that takes the ID of the container and starts the container again and again you can start multiple containers at once if you want like this and they have two containers running now you saw that we use ID of the container in various Docker commands so to start the container to restart it to check the logs Etc but ID is hard to remember and you have to look it up all the time so as an alternative you can also use container name for all these commands instead of the ID which gets auto-generated by Docker but we can actually rewrite that and we can give our containers a more meaningful names when we create them so we can stop those two containers using the ID or the name like this so these are two different containers one with the ID one with name and we're going to stop both of them there you go now when we create a new container we can actually give it a specific name and there is another flag for that which is dash dash name and then we provide the name that we want to give our container let's say this is a web app so that's what we're going to call our container and let's execute if I do Docker PS you see that the name is not some auto-generated random thing but instead our container is called web app so now we can do Docker locks and name of our container like this now we've learned about Docker Hub which is actually what's called a Public Image registry which means those images that we used are visible and available for public but when a company creates their own images of their own applications of course they don't want it to be available publicly so for that there are what's called private Docker Registries and there are many of them almost all Cloud providers have a service for private Docker registry for example AWS is ECR or elastic container registry service Google Azure they all have their own Docker Registries Nexus which is a popular artifact storage service has Docker registry even Docker Hub has a private Docker registry so on the landing page of Docker Hub you saw this get started form so basically if you want to store your private Docker images on Docker Hub you can actually create a private registry on Docker Hub or even create a public registry and upload your images there so that's why I actually have an account because I have uploaded a couple of images on Docker Hub that my students can download for different courses and there is one more concept I want to mention related to registry which is something called a repository which you also often hear Docker repository Docker registry so what is the difference between them very simply explained AWS ECR is a registry so basically that's a service that provides storage for images and inside that registry you can have multiple repositories for all your different application images so each application gets its own repository and in that repository you can store different image versions or tags of that same application the same way dockerhub is a registry it's a service for storing images and on Docker Hub you can have your public repositories for storing images that will be accessible publicly or you can have private repositories for different applications and again you can have repository dedicated for each application so that's a side note there if you hear these terms and Concepts and you know what is the difference between them now I mentioned that companies would want to create their own custom images for their applications so how does that actually work how can I create my own Docker image for my application and the use case for that is when I'm done with development the application is ready it has some features and we want to release it to the end users so we want to run it on a deployment server and to make the deployment process easier once you deploy our application as a Docker container along with the database and other services that are also going to run as Docker containers so how can we take our created deployed application code and package it into a Docker image for that we need to create a definition of how to build an image from our application and that definition is written in a file called a Docker file so that's how it should be called creating a simple Docker file is very easy and in this part we're going to take a super simple node.js application that I prepared and we're going to write a Docker file for that application to create a Docker image out of it and as I said it's very easy to do so this is the application it is extremely simple I just have one server.js file which basically just starts the application on Port 3000 and then it just says welcome when you access it from the browser and we have one package of Json file which contains this dependency but the express library that we use here to start the application super lean and simple and that's the application from which we're going to create a Docker image and start it as a Docker container so let's go ahead and do that so in the root of the application we're going to create a new file called Docker file so that's the name and you see that most code editors actually detect Docker file and we get this Docker icon here so in this Docker file we're going to write a definition of how the image should be built from this application so what does our application need it needs a node installed because node should run our application right so if I wanted to start this application luckily for my terminal I would execute node SRC so the source folder and server.js command to start the application so we need that node command available inside the image and that's where the concept of Base image comes in so each Docker image is actually based on this base image which is mostly a lightweight Linux operating system image that has the node npm or whatever tool you need for your application installed on top of it so for a JavaScript application you would have node base image if you have Java application we will use an image that has Java runtime installed again Linux operating system with Java installed on top of it and that's the base image and we Define the base image using a directive in Docker file called from we're saying build this image from the base image and if I go back to Docker Hub and search for node you will see that we have an image which has node and npm installed inside and base images are just like other images so basically you can pile and build on top of the images in Docker so they're just like any other image that we saw and they also have text or image versions so we're going to choose node image and a specific version and let's actually go for 19-alpine so that's our base image and our first directive in the docker file so again this will just make sure that when our node.js application starts in a container it will have a node and npm commands available inside to run our application now if we start our application with this command we will see that we get an error because we need to First install dependencies of an application we just have one dependency which is press Library which means we would have to execute npm install command which will check the package.json file read all the dependencies defined inside and install them locally in node modules folder so basically we're mapping the same thing that we would do to run the application locally we're making that inside the container so we would have to run npm install command also inside the container so as I mentioned before most of the docker images are Linux based Alpine is a Linux a lightweight Linux operating system distribution so so in Docker file you can write any Linux commands that you want to execute inside the container and whenever we want to run any command inside the container whether it's a Linux command or node command npm command whatever we executed using a run directive so that's another directive and you see that directives are written in all caps and then comes the command so npm install which will download dependencies inside the container and create a node modules folder inside the container before the application gets started so again think of a container as its own isolated environment it has a simple Linux operating system with node and npm installed and we're executing npm install however we need application code inside the container as well right so we need the server.js inside and we need the package.json because that's what npm command will need to actually read the dependencies and that's another directive where where we take the files from our local computer and we paste them copy them into the container and that's a directive called copy and you can copy individual files like package.json from here into the container and we can say where in container on which location in the file system it should be copied to and let's say it should be copied into a folder called slash app inside the container so this is on our machine right we have package.json here this is inside the container it's a completely isolated system from our local environment so we can copy individual files and we can also copy the complete directories so we also need our application code inside obviously to run the application so we can copy this whole Source directory so we have multiple files inside we can copy the whole directory into the Container again in slash app location and the slash at the end is also very important so the docker knows to create this folder if it doesn't exist in the container yet so the roots of Linux file system app folder inside and then slash so now all the relevant application files like package.json and the whole Source directory are copied into the container on this location the next thing we want to do before we can execute npm install command is to actually change into that directory right so in Linux we have this CD right to change into a directory in order to execute the following commands inside the directory in Docker file we have a directive for that called work dear it's a working directory which is an equivalent of changing into a directory to execute all the following commands in that directory so we can do slash app here so it sets this path as the default location for whatever comes afterwards okay so we're copying everything into the Container then we are setting the working directory or the default directory inside the container and then we're executing npm install again within the container to download all the dependencies that application needs that are defined here and finally we need to run the application right so after npm install the node command should be executed and we learned to execute commands we use the Run directive however if this is the last command in the docker file so something that actually starts the process itself the application inside we have a different directive for that called CMD so that's basically the last command in the docker file and that starts the application and the Syntax for that is the command which is node and the parameter gserver.js so we copied everything into slash app so we have the server.js inside the app directory and we're starting it or running it using node commit that's it that is the complete Docker file which will create a Docker image for our node.js application which we can then start as a container so now we have the definition in Docker file it's time to actually build the image from this definition I'm going to clear this up and without changing to the terminal we can actually reuse this one we can execute a Docker command to build a Docker image which is super easy we just do Docker build then we have a couple of options that we can provide the first one is the name of the image so just like all those images have names right like node release Etc and the text we can also name our image and give it some specific tag and we do that using this Dash T option and we can call our application node app maybe with Dash doesn't matter and we can give it a specific tag like 1.0 for example and the last parameter is the location of dockerfile so we're telling Docker build an image with this name with this tag from the definition in this specific Docker file right so this is a location of Docker file in this case we are in the directory where Docker file is located so it's going to be the current directory so this dot basically refers to the current folder where Docker file is located so now if we execute this as you see Docker is actually building the image from our Docker file and it looks like it succeeded where it started building the image you see those steps those directives that we defined here so we have the first one from directive got executed then we have the copy as a second step then we have copy The Source folder setting work directory and running npm install and then the last one just started the application so now now now if I do Docker images in addition to those nginx images we downloaded previously from Docker Hub we should actually see the image that we just created this is the node app image with tag 1.0 and some other information so that's our image and now we can start this image and work with it just like we work with any other image downloaded from Docker Hub so we're going to go ahead and run container from this node app image and make sure that the application inside is actually working so we're going to do Docker run node app image with 1.010 and we're going to pass in parameter to start in detach mode and also we want to expose the port right we want to be able to access the application the node application from localhost and we know that the application inside the container will start on Port 3000 because that's what we have defined here so the application itself will be running on Port 3000 so that's inside container and we can bind it to whatever Port we want on localhost and we can do 3000 the same as in the container so this is the host port and this is container port and now if I execute command and do Docker PS we should see our node app running on Port 3000 and now the moment of truth going back to the browser and opening localhost 3000 there is our welcome to my awesome app message from our application and we can even check the logs by grabbing the ID of our not app and doing Docker blocks with the ID and that's the output of our application inside the container so that's how easy it is to take your application package it into a Docker image using Docker file and then run it as a container and finally going back to this graphical user interface client that Docker desktop actually provides us with now we are able to see other containers and images here as well and that's how this UI actually looks like it gives you a pretty good overview of what containers you have which ones are currently running which ones are stopped with their names and so on and you even have some controls here to start a stop container like this or even stop it again restart container deleted whatever and the same way you have a list of images including our own image and you can also create containers directly from here using some controls so I personally prefer the command line interface to interact with Docker but some feel more comfortable using the visual UI so whichever you prefer you can actually choose to work with either now we've learned a lot of basic building blocks of Docker however it's also interesting to see how Docker actually fits in in the complete software development and deployment process with lots of other Technologies as well so in which steps throughout this whole process is Docker relevant so in this final part of the crash course we're gonna see Docker in big picture view of software development life cycle so let's consider a simplified scenario where you're developing a JavaScript application on your laptop right on your local development environment your JavaScript application uses a mongodb database and instead of installing it on your laptop you download a Docker container from the docker hub so you connect your JavaScript application with the mongodb and you start developing so now let's say you developed the application first version of the application locally and now you want to test it or you want to deploy it on the development environment where a tester in your team is gonna test it so you commit your JavaScript application in git or in some other version control system that will trigger a continuous integration a Jenkins build or whatever you have configured and Jenkins build will produce artifacts from your application so first you will build your JavaScript application and then create a Docker image out of that JavaScript artifact right so what happens to this Docker image once it gets created by Jenkins build it gets pushed to a private Docker repository so usually in a company you would have a private repository because you don't want other people to have access to your images so you push it there and now is the next step could be configured on Jenkins or some other scripts or tools that Docker image has to be deployed on a development server so you have a development server that pulls the image from the private repository your JavaScript application image and then pulls the mongodb that your JavaScript application depends on from a Docker Hub and now you have two containers one your custom container and a publicly available mongodb container running on dev server and they talk to each other you have to configure it of course they talk and communicate to each other and run as an app so now if a tester for example or another developer logs in to a Dev server they will be able to test the application so this is a simplified workflow how Docker will work in a real life development process so in a short time we actually learn all the basic building blocks the most important parts of Docker so you understand what images are how to start containers how they work and how to access them as well as how to actually create your own Docker image and run it as a container but if you want to learn more about Docker and practice your skills even more like how to connect your application to a Docker container learn about Docker compose Docker volumes Etc you can actually watch my full Docker tutorial and if you want to learn Docker in the context of devops and really really Master it with things like private Registries using Docker to run Jenkins integrate Docker in cicd pipelines and use it with various other Technologies like terraform ansible Etc you can check out our complete devops bootcamp where you learn all these and much more

hey I'm Dave welcome to my shop today in Dave's Garage we're going to explore a technology that you've likely heard of but that I argue it's finally time to master Docker I'll show you how to install Docker set it up run Doom spin up a web server create your own custom containers use Docker Hub and much more now imagine that you want to run a particular application so you take a machine you install the operating system and then you install any dependencies for the app that you want to use and then you install the app and you finally get it all configured and you set it up the way that you like but you can't take it with you and you're tied to that machine but now imagine that there's a way to package up all that work so that anyone can launch it on any machine anytime from anywhere well that's Docker Docker is also this episode's sponsor because it just made sense although they didn't get any input into the video and didn't get to see it before it aired so let's nerd out in a way that I doubt they could foresee add it's most basic Docker is a system that allows you to package up all the requirements software and configuration for an application or installation and then bundle it as a standalone container that container holds everything needed to set up and run the application and the container can be run on any system that has Docker installed without any other configuration or installation it just works your containers can even run seamlessly across Windows Linux and Mac Docker takes care of that layer for you whether we expect to be content running other people's pre-made containers or to create our own the first thing we need to do is to install Docker on our system system this will install both the docker client and the docker engine so let's have a quick look at the relatively simple installation process installation varies by System but on the Mac I simply drop it into applications open up applications then as soon as it is copied I can double click it to run it we'll accept the agreement I'll pick the recommended settings I'll sign in and then I'll select of course full stock developer for only exclusive hobby projects and with that Docker and EST demon are up and running with Docker and installed we've essentially gained two major programs Docker doxe the client piece and Docker d. XE the demon or the engine now maybe in the comments you can let me know why in Unix originally and in Linux today background processes are called demons is it because they never die or you can't kill them I don't really get it I'm sure there's a history lesson there but I don't know it so let me know why not call them Forest gumps they're always running anyway regardless of why it's called a demon the docker D process runs in the background at all times and behind the scenes these two components communicate with each other through a rest API every command executed by the docker client will be translated to that rest API this is all behind the scenes and then sent on and processed by the demon we don't need to worry about all this Plumbing any further and so we won't for our purposes we use the dock or client mostly via the command line and it does what it needs to do behind the scenes to fulfill our requests to test our installation let's run a demo container where are we going to get this container or from the web through a service known as Docker Hub it's like a big Library where people can store and share their container images and it's great because you don't have to worry about taking your containers with you you can check it into the docker Hub and then you can run it from any machine that has Docker installed and an internet connection let's try it with the demo container somewhat predictably named hello-world all we need to do is to go to the command line and type in Docker run run run hello-world with that Docker will run off to the web hit the docker hub download the container automatically and then launch it on your machine and if everything is working end to end as it should be we'll get some output from the program welcoming us to the world of Docker this process actually took four steps first the docker client program called the docker demon via that rest API and communicated what it is that we're trying to do which is to run a container then the docker engine pulled the hello world image from the docker Hub created a new container on the machine to host it and executed it and because the output was streamed back across the API to our Docker client we were able to see its output containers represent a significant advancement in software development and management encapsulating much more than just software and configuration they can package an entire runtime environment which includes the application code its dependencies libraries binaries and all necessary configuration files this complete approach ensures that everything needed for the application to run is bundled within the container guaranteeing compatibility and reducing conflicts one of the key features of containers is their ability to provide isolation through Technologies like name spaces and control groups containers create isolated environments for applications separating them and the host from each other namespaces give each container its own distinct view of system resources such as process IDs network interfaces and file systems while cgroups limit and monitor the resources that each container can use such as CPU memory and dis iio this isolation is crucial for maintaining application stability and security preventing ations from ever interfering with one another containers also excel in portability by packaging the application and its dependencies together containers ensure that the application will run consistently across different environments including on Mac Linux and windows whether in development testing or production this consistency eliminates the classic problem of it works on my machine by ensuring the same environment is replicated wherever the container runs furthermore containers are platform independent they can run on any system that supports the container running time whether it's on premises in the cloud or on your developer's laptop there's a great old Docker meme a child developer protests that it works on his machine and the senior developer proclaims then we'll ship your machine and that's how Docker was born probably not but it's a good meme efficiency is another advantage of containers unlike traditional virtual machines which require a full operating system image and can be resource intensive containers share the host systems OS kernel the sharing reduces over overhead making containers much smaller in size and enabling them to start up quickly the rapid startup times of containers facilitate Agile development and deployment processes allowing for Rapid scaling and more efficient resource utilization management and orchestration of containers are streamlined through tools like kubernetes Docker provides a platform for developing shipping and running applications and containers while kubernetes offers an orchestration system for automating the deployment scaling and management of containerized applications these tools are designed to handle the complexity of managing multiple containers across different environments ensuring reliability and scalability and security is another aspect where containers offer benefits containers use features like secure Computing mode or set comp to restrict system calls and apply security policies through app armor or SE Linux profiles controlling the resources and the actions of containers additionally tools for image vulnerability scanning help ensure the container images are free from no inv vulnerabilities before deployment adding an extra layer of security containers provide a comprehensive solution for running applications by encapsulating the entire runtime environment offering isolation portability efficiency and enhanced security this makes containers a powerful tool in modern software development and deployment for exceeding the capabilities of Simply bundling software and configuration together of course as useful as it is to build store and run your own containers Docker Hub gives you access to thousands of pre-built containers that you can execute on demand as a simple example let's run Doom in a container all we need to enter is the docker run command with the ID of the container that we want to run such as Docker run Chasm web Doom 1.14.0 I'm specifying the version because I know that version worked it's almost a gigabyte to download so depending on your network speed it might take a few minutes to download all of the required layers you'll notice that as a container downloads it comes down in layers rather than all at once each layer represents a different set of changes to the file system by separating the container contents into layers Docker can reuse layers across different containers for example if multiple containers use the same base image like kbuntu that base image is downloaded only once and then shared amongst all the containers that use it when the Doom server is done loading and running we need to remember that it's not running on our machine but rather on a container within it connected only by a single Network Port 6901 so to access Doom we browse to Local Host Port 6901 and our game should spring to life well that's all well and good but running Doom doesn't require a lot of setup but let's say we wanted to host a web server if you've ever set up a full web server then you know what a pain it can be in terms of configuration and getting all the files into the right places and installing All the Right Packages and dependencies and so on with Docker we simply run the engine X web server image and it will do everything else there are a few common command line options that we'll use over and over again and since you'll likely make use of them too let's go over a few of them now now the first flag is the dasit flag which specifies that we're going to run the container interactively giving us access to its output the next flag is the- RM flag which tells Docker to remove any existing copy of the container and start fresh we also specify DD to indicates detached so the process will run in the background and finally we use- P to map the network port in the container to one on the host mapping the ports can be a little complicated at first but it simply directs the container where to hook its Network ports into to the real System since the web server container expects to be able to use port 80 for a web server and because the real Port 80 might already be in use it gives us a way to map multiple Network applications in this case we'll have the real port 8080 mapped to the container's port 80 and so the web server will be available on Port 880 and sure enough if we run the container and browse to Port 880 we can see the web server running happily all with a on line command and no additional setup we could run a completely separate independent web server on Port 8081 as well by simply spinning up another container and each would run equally well completely isolated from the other now that we've experienced running a few customized containerized applications it's time to look at finally creating one of our own let's say we wanted to package up a particular application like btop which is a very cool task manager of sorts for Linux that means our container like pretty much all containers these days will be based on a Linux image and we've got to pick which one remember our container will work and run on Mac windows and Linux itself even even though it's based purely on Linux the docker engine will take care of presenting the Linux kernel API to our container how that is implemented behind the scenes varies wildly from Windows to Mac for example but it's behind the scenes so we don't have to worry about it but we do have to pick a starting drro and auntu is the one I'm most familiar with so let's go with that to start building the docker container we need to create a file called Docker file that will describe what goes into to The Container image and the first line in that file will tell Docker what system to base image on now normally everything after the colon refers to a version unless you have a very specific reason to pick a prior working version as we did with doom you can usually generally just specify latest in its place and you'll predictably get the latest version of whatever it is you're asking for that use of the RM flag also forces local deletion of your current copy and forces it to go and get the latest copy if you want to force an update our next step in setting up the container from a bare Linux image will be to install btop on the container in our darker file we specify the Run command followed by the very same commands that we would run on the command line to complete the installation just as we would do manually the container will now do automatically for us our file then ends with a CMD line that specifies the actual command to be run and this is the line that actually runs btop after we've saved our Docker file we go to the command line and we issue a Docker build command which will cause Docker to read our Docker file definition we just created and to build our container and so the command will be Docker build- T btop image and then a period to indicate current directory to build the container image Docker will execute all of our steps including the app to install commands inside of our Docker file and after about 1 to two minutes of processing depending on your network speed if all goes well your container will pop out the other end of the process and be ready to run to execute our local container we use the docker run command Docker run- rm- it btop D image and with that btop Springs into action now the more clever students up in the front row might be wondering why I would install btop in a container so that it runs basically alone and there's nothing else to see now you might think this is just a silly oversight on my part that I just figured out live now but I assure you that what I really wanted to show you is the level of isolation that a container experiences it cannot even see the rest of the system let alone interact with it in any way on Windows it's done via the wsl2 subsystem and a combination of job objects and silos either way the net effect is total isolation for your container fans of the channel might remember a couple of years ago that I wrote a prime Civ that I used to compare the performance of C++ C and python Sunday Sunday Sunday come on down tonight for the event of the decade right here in Dave's Garage you prepare to get annihilated and the amazing the astounding The Unbelievable one night only one night only one night only you are free ticket will get you the whole seat but you'll only need the edge you witness more than 50 computer languages in a showdown so unhinged we almost had to hold it in New Jersey since then under the careful administration of folks like ruer and tutor it has grown like wildfire exploading to several hundred Solutions spanning almost 100 different computer languages this is all on GitHub and if you search for GitHub primes you'll find it the team has built a system where at least once a day all the competing Solutions are compiled linked run and benchmarked with the score for each Lang being recorded in a database that has a web app for easy viewing I'll include a link to the results in the video description but the more interesting part is how we acquire those results each night you might imagine we have a super powerful machine setup that has a 100 different compilers installed and perhaps that's one way to do it if you can get them all to Coexist on the same machine at the same time but it's not the way we went and I should mention it all runs on an Intel core2 Duo under a desk somewhere each language exists in its own Docker container that is configured with only the tools like the compiler and the configuration needed to build that language's project although most of our examples have been built on an Ubuntu base many of the Prime Solutions use leaner dros like Alpine Linux to keep the overall image size down and again if 10 Solutions use the same Alpine Linux layer it only gets downloaded that one time most of the containers are fairly simple but a few like the Assembly Language solution for 6502 or the solution that solves primes in Minecraft are reasonably complex the beauty of it is is that the complexity is compartmentalized along with just the language and tools in a single container none of it spills over and messes up any of the other containers I also make extensive use of Docker in managing some of my own home lab work for example many of the LED effects in the background here such as the colored Windows behind me are run by a server process that uses Wi-Fi to communicate with a series of esp32 chips that server process which has a thread for handling each Target runs entirely within its own container and it can be run on any machine with Docker it even runs on a pi the docker file for my container is not based on a bar Linux installation but upon one provided by Microsoft for running asp.net 6 applications it then descends into the source code folder builds the application publishes it and runs it before our little LED server application can run it needs to know what time it is locally so the second stage of the docker file runs the environment commands needed to set up the time zone and with that it can launch and run our server now if you found today's episode to be any combination of informative or entertaining remember I'm mostly in this for the subs and likes so I'd be honored if you consider subscribing to my channel and leaving a like on the video and if you're already subscribed thank you please do check out the free sample of my new book on Amazon the non-visible part of the autism spectrum it's intended for folks that don't have an ASD diagnosis but who suspect they might have a few characteristics that put them somewhere on the Spectrum it's everything I know now about living a successful life on the spectrum that I wish I know one long ago check it out at the link in the video description in the meantime and in between time I'll see you next time right here in Dave's Garage and that's all I have to say about that

hello and welcome to this complete Docker course by the end of this course you'll have a deep understanding of all the main Concepts and also a great big picture overview of how Docker is used in the whole software development process the course is a mix of animated theoretic explanations but also Hands-On demos for you to follow along so get your first hands-on experience and confidence using Docker in your projects so let's quickly go through the topics I'll cover in this course we will start with the basic concepts of what Docker actually is and what problems it solves also we'll understand the difference between Docker and virtual machine and after installing Docker we will go through all the main Docker commands to start and stop containers debug containers Etc after that we'll see how to use Docker in practice by going through a complete workflow with a demo project so first we'll see how to develop locally with containers then we'll run multiple containers or services with Docker compos we'll build our own Docker image with Docker file and we'll push that built image into a private Docker repository on AWS and finally we'll deploy our containerized application last but not least we'll look at how to persist data in Docker learning the different volume types and afterwards configure persistence for our demo project if you get stuck anywhere just comment under the video and I will try my best to answer your questions also you can join the private Tech worldwi community group on Facebook which is there to exchange your knowledge with others and connect with them if you like the course by the end of the video be sure to subscribe to my channel for more related content so let's get started so we'll talk about what a container is and what problems it solves we will also look at a container repository which is basically a storage for containers we'll see how a container can actually make the development process much easier and more efficient and also how they solve some of the problems that we have in the deployment process of applications so let's dive right into it what a container is a container is a way to package applications with everything they need inside of that package including the dependencies and all the configuration necessary and that package is portable just like any other artifact is and that package can be easily shared and moved around between a development team or development and operations Steam and that portability of containers plus everything packaged in one isolated environment gives it some of the advantages that makes development and deployment process more efficient and we'll see some of the examples of how that works in later slides so as I mentioned containers are portable so there must be some kind of a storage for those containers so that you can share them and move them around so containers live in a container repository this is a special type of storage for containers many companies have their own private repositories where they host or the where they store all the containers and this will look something like this where you you can push all of the containers that you have but there is also a public repository for Docker containers where you can browse and probably find any application container that you want so let's head over to the browser and see how that looks like so if I here search for a dockerhub which is the name of the public repository for Tucker I will see this official website so here if you scroll down you see that there are more than 100,000 container images of different applications hosted or stored in this Docker repository so here you see just some of the examples and for every application there's this official Docker container or Docker container image um but if you are looking for something else you can search it here and I see there's an official image for let's say Jenkins uh but there's also a lot of non-official images or container images that developers or or even from Jenkins itself that they actually store it here so public repository is where you usually get started when you're using or when you're starting to use the containers where you can find any application image so now let's see how containers improve the development process by specific examples how did we develop applications before the containers usually when you have a team of developers working on some application you would have to install most of the services on your operating system directly right for example you are developing some JavaScript application and you need a postp ql and you need red for messaging and every developer in the team would then have to go and install the binaries of those services and configure them and run them on their local development environment and depending on which operating system they're using the installation process will look actually different also another thing with installing services like this is that you have multiple steps of installation so you have a couple of commands that you have to execute and the chances of something going wrong and error happening is actually pretty high because of the number of steps required to install each service and this approach or this process of setting up a new environment can actually be pretty tedious depending on how complex your application is for example if you have 10 services that your application is using then you would have to do that 10 times on each operating system environment so now let's see how containers solve some of these problems with containers you actually do not have to install any of the services directly on your operating system because the container is its own isolated operating system layer with Linux based image as we saw in the previous slides you have everything packaged in one isolated environment so you have the posis ql with a specific version packaged with the configuration and the start script inside of one container so as a developer you don't have to go and look for the binaries to download on your machine but rather you just go ahead and check out the container repository to find that specific container and download on your local machine and the download step is just one Docker command which fetches the container and starts it at the same time and regardless of which operating system you're on the command the docker command for starting the container will not be different it will be exactly the same so if you have 10 applications that your JavaScript application uses and depends on you would just have to run 10 Docker commands for each container and that will be it which makes the setting up your local development environment actually much easier and much more efficient than the previous version also as we saw in the demonstration before you can actually have different versions of the same application running on your local environment without having any conflict so now let's see how containers can improve the deployment process before the containers a traditional deployment process will look like this development team will produce artifacts together with a set of instructions of how to actually install and configure those artifacts on the server so you would have a jar file or something similar for your application and in addition you would have some kind of a database service or some other service also with a set of instructions of how to configure and set it up on the server so development team would give those artifacts over to the operations team and the operations team will handle setting up the environment to deploy those applications now the problem with this kind of approach is that first first of all you need to configure everything and install everything directly on the operating system which we saw in the previous example that could actually lead to conflicts with dependency versions and multiple Services running on the same host another problem that could arise from this kind of process is when there is misunderstanding between the development team and operations because everything is in a textual guide as instructions so there could be cases where developers forget to mention some important point about configuration or maybe when operations team misinterpret some of those instructions and when that fails the operations team has to go back to the developers and ask for more details and this could lead to some back and forth communication until the application is successfully deployed on the server with containers this process is actually simplified because now you have the developers and operations working in one team to package the whole configuration dependencies inside the application just as we saw previously and since it's already encapsulated in one single environment and you don't have to configure any of this directly on the server so the only thing you need to do is run a Docker command that pulls that container image that you've stored somewhere in the repos repository and then run it this is of course a simplified version but that makes exactly the problem that we saw on the previous slide much more easier no environmental configuration needed on the server the only thing of course you need to do is you have to install and set up the docker runtime on the server before you will be able to run containers there but that's just onetime effort now that you know what a container concept is let's look at what a container is technically so technically container is made up of images so we have layers of stacked images on top of each other and at the base of most of the containers you would would have a Linux based image which is either Alpine with a specific version or it could be some other Linux distribution and it's important for those base images to be small that's why most of them are actually Alpine because that will make make sure that the containers stay small in size which is one of the advantages of using container so on top of the base image you would have application image and this is a simplified diagram usually you would have these intermediate images that will lead up to the actual application image that is going to run in the container and of course on top of that you will have all this configuration data so now I think it's time to dive into a practical example of how you you can actually use a Docker container and how it looks like when you install it and download it and run it on your local machine so to give you a bit of an idea of how this works let's head over to dockerhub and search for postris ql so here which is a Docker official image I can see some of the versions and let's say say I'm looking specifically for older version I don't know 96 something so I'm going to pull that one so this is a dock repository so that I can actually go ahead and pull the containers from that repository directly and because it's a public repository I don't have to log into it I don't have to provide any authentication credentials or anything I can just get started with simple Docker command without doing or configuring anything to access dockerhub so on my terminal I can just do Docker Pole or I can even do Docker run and then just copy the the image name and if I don't specify any version it will just give me the latest but I want a specific version so I'm just I'm going to go with 9.6 actually just to demonstrate so I can provide the version like this with a column and I can start run so as you see the first line says unable to find image locally so it knows that it has to go to dockerhub and pull it from there and the next line says pulling from library postgress and here you see a lot of hashes that says downloading and the this is what I mentioned earlier which is Docker containers or any containers are made up of layers right you have the Linux image layer you have the application layers and so on so what what you see here are actually all those layers that are separately downloading from the dockerhub on my machine right and the advantage of splitting those applications and layers is that actually for example if the image changes or I have to download a newer version of pogress what happens is that the layers they're the same between those two applications two versions of posis will not be downloaded again but only those layers that are different so for example now it's going to take around 10 or 15 minutes to download this one image because I don't have any pogress locally but if I were to download the next version it will take a little bit less time because some layers already exist on my local machine so now you see that it's already logging because it this command that I ran here the docker run with the container name and version it fetches or it pulls the the container but it also starts it so it executes the start script right away as soon as it downloads it and here you see the output of the starting of the application so it just gives some output about um starting the server and doing some configuration stuff and here you see database system is ready to accept connections and launcher started so now let's open the new tab and see with Docker PS command you can actually see all the running containers so here you see that postgress 96 is running and it actually says image so this is another important thing to understand when we're talking about containers there are two technical terms image and a container and a lot of people confuse those two I think and there is actually a very easy distinction between the two so image is the actual package that we saw in one of those previous slides so the application package together with the configuration and the dependencies and all these things this is actually the artifact that is movable around is is actually the the the image container is when I pull that image on my local machine and I actually started so the application inside actually starts that creates the container environment so if it's not running basically it's an image it's just an artifact that's lying around if I start it and actually run it on my machine it is a container so that is the distinction so here it says the active running containers with a container ID image that it's running from and some entry commands that it executed and some other status information so this means that poql is now running on my local machine simple as that if I were now to uh need let's say another version of pogress to run at the same time on my local machine I could just go ahead and say let's go back and let's say I want want to have 9.6 and 10.10 running at the same time on my local machine I just do run postgress and run again it doesn't find it locally so it pushes and this is what I actually explained to you earlier because it's the same application but with just a different version version some of the layers of the image are the same so I don't have to fetch those again because they are already on my machine and it just fetches the layers that are different so that saves a little bit of uh time and I think it's it could be actually good Advantage so now we'll wait for other image layers to load so that we have the second postgress version running and now you see I have postgress 9.6 running in this uh command line tab and I have postgress version 10.10 running in the next one so I have two postes with different versions running and I can actually output them here have both of them running and there's no conflict between those two like I can actually run any number of applications with different versions maybe of the same application with no problem at all and we going to go through how to use those containers in your application and the port configuration and some of the other configuration stuff later in this tutorial when we do a deep dive but this is just for you to get the first visual image of how Docker containers actually work how they look like and how easily you can actually start them on your local machine without having to implement lement a specific version of postgress application and do all the configuration yourself when I first started learning Docker after understanding some of the main Concepts my first question was okay so what is the difference between Docker and an Oracle uh virtual box for example and the difference is quite simple I think and in this short video I'm I'm going to cover exactly that and I'm going to show you the difference by explaining how docu works on an operating system level and then comparing it to how virtual machine works so let's get started in order to understand how Docker works on the operating system level let's first look at how operating system is made up so operating systems have two layers operating system kernel and the applications layer so so as you see in this diagram the kernel is the part that communicates with the hardware components like CPU and memory Etc and the applications run on the Kernel layer so they are based on the Kernel uh so for example we you all know Linux operating system and there are lots of distributions of Linux out there there's buonto and Debian and there's Linux Mint Etc there are hundreds of distributions they all look different so the graphical user the interface is different the file system is maybe different so a lot of applications that you use are different because even though they use the same Linux kernel they use different or they Implement different applications on top of the kernel so as you know Docker and virtual machine they're both virtualization tools so the question here is what parts of the operating system they virtualize so docker virtualizes the applications layer so when you download a Docker image it actually contains the applications layer of the operating system and some other applications installed on top of it and it uses the kernel of the host because it doesn't have its own kernel the virtual box or the virtual machine on the other hand has the applications layer and its own kernel so it virtualizes the complete operating system which means that when you download a virtual machine image on your host it doesn't use your host kernel it boots up its own so what does this difference between Docker and virtual machine actually mean so first of all the size of Docker images are much smaller because they just have to implement one layer so Docker images are usually couple of megabytes uh virtual machine images on the other hand can be couple of gigabytes large a second one is the speed so you can run and start Docker containers much faster than the VMS because they every time you start them you have they have to put the operating system kernel and the applications on top of it the third difference is compatibility so you can run a virtual machine image of any operating system on any other operating system host but you can't do that with Docker so what is the problem exactly let's say you have a Windows operating system with a kernel and some applications and you want to run a Linux based Docker image on that Windows host the problem here is that a Linux based Docker image might not be compatiable with the windows kernel and this is actually true for the windows versions below 10 and also for the older Mac versions which if you have seen how to install Docker on different operating systems you see that the first step is to check whether your host can actually run Docker natively which basically means is the kernel compatible with the docker images so in that case a workaround is that you install a technology called Docker toolbox which abstracts away the kernel to make it possible for your host to run different Docker images so in this video I will show you how to install Docker on different operating systems the installation will differ not only based on the operating system but also the version of that operating system so you can actually watch this video selectively depending on which OS and the version of that OS you have I will show you how to find out which installation step applies to you in the before installing section which is the first one so once you find that out you can actually directly skip to that part of the video where I explained that into details I will put the minute locations of each part in the description part of the video and also I will put all the links that I use in the video in the description um so that you can easily access them also if you have any questions during the video or if you get stuck installing the docker on your system please post your question or problem in the comment section so that I can um get back to you and help you proceed or maybe someone from the community will uh so with that said let's let's dive right into it so if you want to install Docker you can actually Google it and you get an official documentation of Docker um it's important to note that there are two additions of Docker there is a community and Enterprise additions um for us to begin with Community Edition will be just fine in the docker Community Edition uh tab there there's a list of operating systems and distributions in order to install docker so for example if we start with Mac we can click in here and we see the documentation of how to install it on Mech which is actually one of the easiest but we'll see some other ones as well so before you install Docker on your Mac or Windows computer there are prerequisites to be considered so for mac and windows there has to be some criteria of the operating system and the hardware met in order to support running Docker if you have Mech go through the system requirements to see if your U Mech version is actually supporting Docker if you have Windows then you can go to the windows Tab and look at the system requirements there or what to know before you install for example one thing to note is that Docker natively runs only on Windows 10 so if you have a Windows version Which is less than 10 then Docker cannot run natively on your computer so if your computer doesn't meet the requirements to run Docker there is a workaround for that which is called Docker toolbox instead of Docker you basically just have to install a Docker toolbox that will become a sort of a bridge between your operating system and the docker and that will enable you to run Docker on your leg Legacy computer so if that applies to you then skip ahead in this video to the part where I explain how to install Docker toolbox on Mac and on windows so let's install Docker for Mac as you see here there are two um channels that you can download the binaries from or the application from we will go with the stable Channel and other things to can see if you have an older version of Mech either software or the hardware please go through the system requirements to see if you can actually install Docker so here there is a detailed description of what make version you need um to be able to run Docker and also you need at least four gab of RAM and by installing Docker you will actually have the whole package of it which is a Docker engine uh which is important or which is necessary to run the docker containers on your laptop the docker command line client which will enable you to execute some Docker commands Docker compose if you don't know it yet don't worry about it but it's just technology to orchestrate if you have multiple containers um and some other stuff that we're not going to need in this tutorial but you will have everything in a package installed so go ahead and download the stable version well I already have Docker installed from The Edge channel so I won't be installing it again but it shouldn't matter because the steps of installation are the same for both so once the docker DMG file is downloaded you just double click on it and it will pop up this window just drag the docker whale app into the applications and it will be installed on your Mach as the next step you will see Docker installed in your applications so you can just go ahead and start it so as you see the docker sign or icon is starting here if you click on it you see the status that Docker is running and you can configure some preferences and check the docker version and so on and if you want to stop Docker or quit it on your Mech you can just do it from here um an important maybe interesting note here is that if let's say you download or install Docker and you have uh more than one accounts on your laptop you will actually get some errors or conflicts if you run Docker at the same time or multiple accounts so what I do for example is that if I switch to another account where I'm also going to need Docker I quit it from here and then I start it from other account so that I don't get any errors so that might be something you need to consider if you use multiple accounts so let's see how to install Ducker for Windows the first step as I mentioned before is to go to the before you install section and to see that your operating system and your computer meets all the criteria to run Docker natively so if you're installing Docker for the first time don't worry about most of these parts like Docker toolbox and Docker machine there are two things that are important one is to double check that your windows version is compatible for Docker and the second one is to have virtualization enabled virtualization is by default always enable abled um other than you manually disabled it so if you're unsure then you can check it by going to the task manager performance CPU Tab and here you can see the status of the virtualization so once you have checked that and made sure that these two prerequisites are met then actually you can scroll up and download the windows installer for from the stable Channel once they install installer is downloaded you can just click on it and follow the installation wizard to install Docker for Windows once the installation is completed you have to explicitly start Docker because it's not going to start automatically so for that you can just go and search for the docker for Windows app on your windows just click on it and you will see the docker whale icon um starting and if you click on that icon you can actually see the status that says stalker is now up and running so this is basically it for the installation now let's see how to install Docker on different Linux distributions and this is where things get a little bit more complicated so first of all you see that in this menu on the on the left you see that for different Linux distributions the installation steps will differ but also for example if we just click on auntu for the guide you can see that in the prerequisites section there is also differentiation between the versions of the same Linux distribution and there may be some even more complicated scenarios where the combination of the version of the distribution and the architecture it's running in um also makes some difference into how to set up Docker on that specific environment because of that I can't go through a Docker installation process of every Linux environment because there are just too many combinations so instead what we'll do is just go through a general overview of the steps and configuration process to get Docker running on your Linux environment and you can just adjust it then for your specific setup so these are some general steps to follow in order to install Docker on your Linux Linux environment first of all all you have to go through the operating system requirements part on the relevant Linux distribution um that applies for you a second step in the documentation to is to uninstall old versions however if it's the first time you installing Docker then you don't have to worry about that you also don't have to worry about the supported storage drivers and you can skip ahead to the part of installing Docker Community Edition so for any Linux distribution here the steps will be or the options for installing Docker will be the same so first option is basically to set up a repository and download the docker from and install it from the docker repository um the second option is to install the packages manually however I wouldn't recommend it and I think the documentation doesn't recommend it either because then you'll have to do a lot of steps of the installation and the maintenance of the versions manually so I wouldn't do that the third one is just for the testing purposes it may be enough for the development purposes as well but I would still not do it which is basically just downloading some automated scripts that will install and setup Docker on your Linux environment however again I wouldn't go with it I would actually just do the first option which is just download Lo in the docker from the repository so in order to install Docker using the first option which is downloading it from the docker repositories you have two main steps so the first one is to set up the repository uh which differs a little bit depending on which distribution you have and then install the docker CE from that repository so from abunto and Debian the steps for setting up the repository are generally just updating your app package then setting up an https connection with the repository and adding the docker's official gpg key which only aono and dbn need you don't have to do this um steps for SOS and Fedora they just have to install the required packages and the last step uh for setting up the repository is basically setting up the stable repository of Docker which we saw previously on the overview that there are two channels which is a stable and Edge here you always have to set up the stable repository optionally you can also set up the edge repository but I'll just do uh stable this time and here also something to notice depending on architecture you have to actually set it or you have to set that as a parameter when you set up the repository so if you have for example a different architecture you can use those steps to display the correct command for it and um I guess that applies to other Linux distributions as well like for example here you also have the second tab where you see a separate command for it so these steps should actually um set up the repository so that as a Next Step you can then install the docker C from those repositories so installing Docker from the setup repository is actually pretty straightforward the steps are same for or similar to all the distributions basically just update the app package and then you just say install Docker CE so this command will just download the latest Docker version if you want to install a specific one which you will need to do in a production environment then you can just uh provide a version like like this to just say Docker minus C equals some specific versions and using this command you can actually look up what versions are available in that repository that you just and with this command actually Docker will be installed um on your Linux environment and then you can just verify using PSE sudo Docker run hello world which is this demo image of Docker you can verify that Docker is running and this will start hello world Docker container on your environment so as I mentioned previously for environments um that do not support running Docker natively there is an workaround which is called Docker toolbox so Docker toolbox is basically an installer for Docker environment setup on those systems so this is how to install uh Docker toolbox on your Mac um this is the whole package that comes with the installation of Docker toolbox which is basically the docker command line Docker machine Docker compose basically all the packages that we saw in the native installation and in on top of that you also get the Oracle VM virtual box so in order to install the docker toolbox it's actually pretty straightforward on this website you can go to the toolbox releases where we have all the leas of latest releases you just take the uh latest release and here you see two assets this one is for Windows obviously and you just download the package for mac and once it's downloaded you just click on it and go through the installation wizard leave all the options by default as they are do not change anything and after the installation you can just validate that the installation is successful and you can actually run Docker so so after seeing the installation was successful screen just go and look up in your launch pad dock quick start terminal and once you open it you should be able to run uh Docker commands and you can just try Docker run hello world which should just start up or bring up um this hello world Docker container on your environment so now let's see how to install Docker toolbox on Windows here see that you get the whole package of Docker Technologies with a toolbox which are basically the same package which you get on the uh Native Docker installation and on top of that you get Oracle VM virtual box which is the tool that enables Docker to run on an older system so before you install Docker tool books you have to make sure that you meet some of the preconditions number one you have to make sure your Windows system supports virtualization and that virtualization must be enabled otherwise Docker Docker won't start so depending on which Windows version you have looking up or checking the virtualization status will be different so I just suggest you Google it and look it up of how to find the virtualization status to see that it's enabled once you have that checked also make sure that your Windows operating system is 64 bits so if those two criteria are met then you can go ahead and install the Locker toolbox the place where you see the releases or the release artifacts is toolbox releases link here which I have open so it's basically a list of the releases you just take the latest one which has two artifacts this is the one for Windows you just download this executable file click on it and go through the installation wizard once the installation is completed there are just couple of steps here you can verify that Docker was installed or the toolbox was installed by just looking up the docker quick start terminal on your windows that app must be installed and once you click on it and open it you should be able to run Docker commands in the terminal so the basic Docker command that you can test will be Docker run hello world which will just fetch this basic uh Docker container from the public registry and run it on your computer if that command is successful it means that Docker was successfully installed on your computer and now you can proceed with the tutorial so in this video um I'm going to show you some basic Docker commands at the beginning I'm going to explain what the difference between container and images because that's something a lot of people confuse then very quickly go through version and tag and then show you a demo of how to use the basic Docker commands um commands that will be enough to pull an image locally to start a container to configure a container and even debug the container so with that said let's get started so what is the difference between container and image mostly people use those terms interchangeably but actually there is a fine difference between the two to see theoretically container is just the part of a container runtime so container is the running environment for an image so as you see in this graphic the application image that runs the application could be postgress redis some other application needs let's say a file system where it can save the log files or where it can store some configuration files it also needs some environmental configuration like environmental variables and so on so all of this environmental stuff are provided by by by container and container also has a Port that is binded to it uh which makes it possible to talk to the application which is running inside of a container and of course it should be noted here that the file system is virtual in container so the container has its own abstraction of an operating system including the file system and the environment which is of course different from the file system and environment of the host machine so in order to see the difference between container and image in action let's head over to the docker Hub and find for example a rice image another thing is that dockerhub all the artifacts that are in the docker Hub are images so we're not talking about containers here all of these things are images Docker official image so we're going to go ahead and pull a rad image out of the doah Hub to my laptop so you see the different layers of the image are downloaded and this will take a couple of of of minutes so once the download is complete I I I can check all the existing images on my laptop using Docker images command so I see I have two images radi and postgress with text image IDs and so on another important aspect of images is that they have texts or versions so for example if we go back to the docker Hub each one each image that you look up in the docker Hub uh will have any different versions the latest is always the one that you get when you don't specify the version of course if you have a dependency on a specific version you can actually choose the version you want and specified and you can select one from here so this is what you see here the tag is basically the version of the image so I just downloaded the latest and I can also see the size of the image so now to this point we have only worked with images there is no container involved and there is no redish running so now let's say I need red running so that my application can connect to it I'll have to create a container of that redice image that will make it possible to connect to the redis application and I can do it by running the redis image so if I say Docker Run Red this will actually start the image in a container so as I said before container is a running environment of an image so now if I open a new tab and do Docker PS I will get status of all the running Docker containers so I can see the container redis is running with a container ID based on the image of redis and some other information about it for example the port that it's running on and so on so as you see here the docker run redis command will start the redis container in the terminal um in an attached mode so for example if I were to terminate this with the control C you see that red application stops and the container will be stopped as well so if I do Docker PS again I see that no container is running so there is an option for Docker run command that make makes it able makes it possible to run the container in a detached mode and that is minus D so if I do dock run minus D redis I will just get the ID of the container as an output and the container will stop running so if we check again Docker PS I see the container with the ID starting with 838 which is the same thing here is running so this is how you can start it in the detached mode now for example if you would want to restart a container because I don't know some uh the application crashed inside or some error happened so you want to restart it you would need the doc container ID so just the first part of it not the whole string and you can simply say Docker stop ID of the container and that will stop the docker container nothing running if you want to start it again you can use the same ID to start again so let's say you stop Docker container at the end of the day you go home you come back the next day open your laptop and you want to restart the stopped container right so if you do Docker PS there is uh the output is empty you don't see any containers so what you can do alternative to just looking up your history command line history is you can do Docker PS minus a which will show you all the containers which uh are running or not running so here you see the container ID again and you can restart it okay so let's try another thing let's say you have two parallel applications that both use redish but in different versions so you would need two redish containers with different image versions running on your laptop right at different times maybe or at the same time so here we have the latest one which is radius 56 and let's head over to the dockerhub and select uh version let's say you need version 4 o so remember the first time that we downloaded the redis image we did Docker pull redis um however if you run Docker if you use Docker run with redice image and the tech which was was was 4.0 it will pull the image and start the container right away after it so it does two commands basically in one so it's Docker pole that Docker start in one one command so if I do this it says it can't find the image locally so it goes and pulls the image from the repository to my laptop and again we see some layers are downloaded and the container is started right away and now if I do Docker PS you see that I have two radices running so this is where it gets interesting now how do you actually use any container that you just started so in this output we you also see the ports section which specifies on which Port the container is listening to the incoming requests so both containers open the same port which is what was specified in the image so in the logs of the container you can see the information running mode stand loone Port 6379 so how does that actually work and how do we not have conflicts while both are running on the same port so to explain that let's head over to our slide and see how this works as you know container is just the virtual environment running on your host and you can have multiple containers running simultaneously on your host which is your laptop PC whatever you working on and your laptop has certain ports available that you can open for certain applications so how it works is that you need to create a so-called binding between a port that your laptop your host machine has and the container so for example in the first container part here you see container is listening on Port 5000 and you bind your laptops Port 5,000 to that containers now you will have conflict if you open two 5,000 ports on your host because you will get a message the port is already bound or is already in use you can do that um however you can have two containers as you see the second and third containers are both listening on Port 3000 which is absolutely okay as long as you bind them to two different ports from your host machine so once the port binding between the host and the container is already done you can actually connect to the running container using the port of the host so in this example URI you would have some app Local Host and then a port of the host and the host then will know how to forward the request to The Container using the port binding so if we head back here you see that containers have their ports and they're both running on the same one however we haven't made any binding between my laptop's port and the container port and because of that the container is basically unreachable by any application so I won't be able to use it so the way we actually do that is by specifying The Binding of the ports during the Run command so I'm going to break this and check that there is just one container running now I'm going to stop the other one as well so we can start the menu okay so we see both containers are here so now we want to start them using The Binding between the host and the container ports but again we have two Rices so we need to bind them to two different ports on my laptop so the way to do it is you do Docker run and you specify with minus P the port of the host that's the first one so let's go with 6,000 it doesn't really matter in this case and the second one is the port that you're binding this two which is the container Port so we know the container Port will be be be 6379 and this is where we bind our so my laptop's port 60002 and if I do this I forgot ra here so and now if I do Docker PS let's actually clean this Docker PS again here you see The Binding here all right so your laptops 6,000 Port is bound to the containers 6 37 9 so now let's do another thing and let's start it in a detached mode so like this let's check again it's running again and now I want to start the second container let's clear this again so here you see it created a bunch of containers because uh when I specified different options with the port binding it actually created new containers um that's why you see a couple of more here so I'm going to copy the image name with the tag for uh. o minus P so for example if I were to do this now now now and I would try to run the other redis the second redis container with the same port on my laptop I would get an error saying Port is already allocated so I can do 60001 and run it again I'll run it in detached mode so that I'm see port and if I go go over here and say Docker PS I see that I have two different radi versions running both of them bound to different ports on my laptop and the containers themselves listening to request on the same port so so far we have seen a couple of basic Docker commands we have seen Docker pull which pulls the image from the repository to local environment we also saw run which basically combines Docker pull and Docker start pulls the image if it's not locally available and then starts it right away then we saw Docker start and Docker stop which makes it possible to restart a container if um you made some changes and you want to um create a new version which makes possible to restart a container if you need to um we also saw docu run with options the one option that we saw was D minus D which is detach so you can run the container in DET detached mode so you can use terminal again minus P allows you to bind Port of your host to The Container so very important to remember minus P then comes the port of your host and then comes the port of your um container whatever it might be we also saw doap PS doap PS minus a which basically gives you all the containers no matter if they're running currently or not we also saw Docker images which gives you all the images that you have um locally so for example if after a couple of months you decide to clean up your space and get rid of some stale images you can actually check them check the list and then go through them and uh delete them you can do the same with stale Docker containers that you don't use anymore or you don't need anymore you can also get rid of them so the final part of the docker basic commands are commands for troubleshooting which are very very useful if something goes wrong in the container you want to see the logs of the container or you want to actually get inside of container get the terminal and execute some commands on it so let's see Docker PS we have two containers running right now we don't have any out we don't see any locks here so let's say something um happens your application cannot connect to redies and you don't know what's happening so ideally you would want to see what logs radi container is producing right the way to do that is very easy you just say Docker locks and you specify the container ID and you see the locks you can also do the lock locks if you don't want to uh remember the container ID or to Docker PS all the time you can remember the name of the container and you can get the logs using the name so a little side note here um as we're talking about the names of the containers so here as you see when a container is created you just get some random name like this so you can name your containers as you want um using another option of the docker run which might be pretty useful sometimes if you don't want to work with the container IDs and you just want to remember the names um or if you just want to differentiate between the containers so for example let's create a new container from r4.0 image using a different name that we choose so I'm going to stop this container and I'm going to create a new one from the same image so let's run it in the detached mode Let's Open the port th000 1 2 6 3 7 9 and give the name to the container and let's call since it's the older um version let's call it red older and we need to specify the image so remember this will create a new container since we're running the docker one command again so if we execute this and check again we see the redis 4.0 image based container is created which is um a fresh new you can see in it created and the name is red is older and we can do the same for the other container so that we kind of know which uh container is what so I'll stop this one and I will use the same command here this will be the latest and I will call this radius latest and since find another Port so I'm going to run it and let's see so here I have two containers running now I know R is older R is latest so for example if the older version has some problems I can just do logs R is older and I can get get my locks so another very useful command in debugging is Docker exit so what we can do with Docker exit is we can actually get the terminal of a running container so let's check again we have two containers running and let's say there is some problem with the latest ready latest container and I want to get a terminal of that container and to maybe navigate a directory inside check uh the lock file or maybe check the configuration file or uh print out the environmental variables um whatever so in order to do that we use Docker exit command with minus t which stands for interactive terminal then I specify the container ID and I say in so I get the B and here you see that the the cursor changed so I'm in inside of the container as a root user and here if I say LS okay the data is empty I can also print out in which directory I am I can go to home directory see what's there um so I have my virtual file system inside of a container and here I can um navigate the different directories and I can check stuff I can also print all the environmental variables to see that something is set correctly um and do all kinds of stuff here and this could be really useful if you have a container with some complex configuration or if for example you are running your own application that you wrote in a container H and you have some complex configuration there um or some kind of setup and you want to validate that everything um is correctly set in order to exit the terminal you just do exit and you're out you can also do the same using the name again if you don't want to work with the IDS and you just want to remember the names of the container to make it easier you can do it with the name as well same thing since most of the container images are based on some lightweight Linux distributions you wouldn't have much of their Linux um commands or applications installed here for example you wouldn't have a curl or some other stuff so you were a little bit more limited in that sense so you can execute a lot of stuff from the docker containers for most of the debugging work um it should be actually enough so the final part to review the difference between Docker run and Docker start which might be confusing for some people let's revisit them so basically Docker run is where you create a new container from an image so Docker run will take an image with a specific version or just latest right as option or as an attribute with Docker start you not working with images but rather with containers so for example um as we saw Docker run has a lot of options you specify with minus D and minus P the port binding and then you have this name of the container and all this stuff so basically you tell Docker at the beginning what kind of container with what attributes name and so on to create from a specific image but once the container is created and you can see that using a con uh the command so for example here the last ones that we created and if you stop it and you want to restart it you just need to use the command do start and specify the container ID and when you start it the container will retain all the attributes that we defined when creating the container using Docker run so Docker run is to create a new container Docker start is to restart a stopped container so once you've learned the docker basic concepts and understood how it works uh it's important to see how Docker is actually used in practice so in software development workflow you will know you have this uh classical steps of development and continuous delivery or continuous integration uh and then eventually gets deployed on some environment right could be a test environment develop environment so it's important to see how Docker actually integrates in all those steps so in the next couple of videos I'm going to concentrate exactly on that so we're going to see some overview of the flow and then we're going to zoom in on different parts and see how Docker is actually used in those individual steps so let's consider a simplified scenario where you are developing a JavaScript application on your laptop right on your local development environment your JavaScript application uses a mongodb database and instead of installing it on your laptop you download a Docker container from the docker Hub so you connect your JavaScript application with the mongodb and you start developing so now let's say you develop the application first version of the application locally and now you want to test it or you want to deploy it on the uh development environment where a tester in your team is going to test it so you commit your JavaScript application in git or in some other version control system uh that will trigger a continuous um integration a Jenkins build or whatever you have configured and Jenkins build will produce artifacts from your application so first it will build your JavaScript application and then create a Docker image out of that JavaScript artifact right so what happens to this Docker image once it gets created by Jenkins build it gets pushed to a private Docker repository so usually in a company you would have a private repository because you don't want other people to have access to your image images so you push it there and now as a Next Step could be configured on Jenkins or some other scripts or tools that Docker image has to be deployed on a development server so you have a development server that pulls the image from the private repository your JavaScript application image and then pulls the mongodb that your JavaScript application depends on from a dockerhub and now you have two containers one your custom container and a publicly available mongodb container running on dev server and they talk to each other you have to configure it of course they talk and communicate to each other and run as an app so now if a tester for example or another developer logs in to a Dev server they be they will be able to test the application so this is a simplified workflow how Brer will work in a real life development process in the next videos I'm going to show you Hands-On demo of how to actually do all this in practice so in this video we are going to look at some practical example of how to use Docker in a local development process so what we're going to do is a simple demo of a JavaScript and nodejs application in the back end to simulate the local development process and then we're going to connect it to a Docker container with a mongodb database in it so let's get started so in this video we're going to see how to work with Docker containers When developing applications so the first step will be is we're going to develop a very simple UI backend uh application using JavaScript very simple HTML structure and nodejs in the back end and in order to integrate all of this in the database we are going to use a Docker container of a mongodb database and um also to make working with the mongodb much easier so we don't have to execute commands in in the terminal we're going to deploy a Docker container of a UI which is called the Express where we can see the database structure and all the updates that our application is making in the database so this development setup should give you an idea of um how Docker containers are actually used in development process so I've already prepared some very simple JavaScript application um so in order to see the code basically we have this index HTML that is very simple code and we have some JavaScript here and we're using nodejs backend that just serves that index HTML file and listens on Port 3000 so we have the server running here in the back end and we have the UI that looks like this so basically it's just a user profile page with some user information and user can edit their profile so if I for example change the name here um and if I change the email address and do changes like this I can save my profile and I have my updates here um however if I refresh the page of course the changes will be lost because it's just JavaScript no JS so there is no persistent compon component in this application so in order to have that which is actually how real life applications work you'll know that you need to integrate the application with a database so using that example I will try to Showcase you how you can actually use the docker containers to make the development process Easier by just pulling one of the databases and attaching it or connecting it to the application so in this case we're going to go with the mongodb application and uh in addition to mongodb contain container we're going to also deploy a mongodb UI which is its own container it's called Express where we can manage or see the database insights and updates from our application much easier so now let's see how that all works so in order to get started let's go to dockerhub and find our uh mongodb image so here let's go to and we have mongodb here actually and the Express which is another dock container that we're going to use for the UI so first let's pull the mongodb official image so I I already have mongodb latest so pulling doesn't take longer on my laptop but you're going to need a couple of seconds probably and the next one we're going to pull is the docker Express which I also have I believe so let's see yes it's also fast so if I check locally I have mongod TB and Express images so the next step is to run both and Express uh containers in order to make the mongod DB database available for our application and also to connect the Express with the mongod DB container so let's do the connection between those two first in order to do that we have to understand another Docker concept Docker Network so how it works is that Docker creates its isolated Docker Network where the containers are running in so so when I deploy two containers in the same docken network in this case and Express they can talk to each other using just the container name without Local Host port number Etc just the container name because they're in the same network and the applications that run outside of doer like our nodejs which just runs from node server is going to connect to them from outside or from the host using Local Host and the port number so later when we package our application into its own Docker image what we're going to have is again Docker network with mongodb container Express container and we're going to have a nodejs application which we wrote including the index HTML and JavaScript for front end in its own doc container and it's going to connect to the mongodb and the browser which is running on the host but outside the docker network is going to connect to our JavaScript application again using host name and the port number so Docker by default already provides some networks so if we say Docker Network LS we can already see these autogenerated Docker networks so we have four of them with different names and the drivers we're not going to go into details here but what we're going to do is create its own network for the mongodb and the Express and we're going to call it Network so let's do this right away going to say Docker Network create and we are going to call it Network so now if I do dock Network LS again I see my docken network has been created so now in order to make our m B container and the Express container run in this Network we have to provide this network option when we run the container in the docker run command so let's start with the so we all know that Docker run is the command to start a container from an image right so we have Docker run which is the basic Docker run command however in this case we want to specify a couple of things um as you learned from the previous videos you have to specify something called Port so we need to open a port of mongodb the default Port of mongodb is 27,7 so we will take that Port actually for both host and container so will run at this port inside of a container and we open the same port on the host so that will take care of the port then we will run it in a detach mode in addition to that there are a couple of things that we can specify when starting up the container and these are environmental variables of the mongot TB let's see um in the official image description you actually have couple of documentation about how to use the image which is very helpful to kind of understand what kind of configuration you can uh apply to it here you see some environmental variables so basically on Startup you can Define what the root username and the password will be which is very um handy because we're going to need those two for the express to connect to the and you can also specify the init database we're just going to provide the username and password because we can create the database from the Express UI later so let's do that and the way you can specify the environmental variables you can actually see here as well is by just let's copy this one so here you say environmental variable that's what the minus E Flex stands for and root username we'll say it's admin and another variable which is the password will be just password so in this way we can actually overwrite what the default username and password will be so two more things that we need to configure in this uh command our container name because we're going to need that container name to connect with the Express so we'll call this one DB let's say and another one we need is the network that we created which was called Network so in order to make this command a little bit more structured do it on multiple lines so let's see so it's more readable so basically all the options or all these flags that we set um to go one more time through them it it's going to start in detached mode uh we are opening the port on the host um username and password that we want mongodb to use uh in the startup process we're going to rewrite or overwrite the name of the container and this container is going to run in a Network and this should actually start the container okay so if you want to see whether it was successful we can log the container and see what's happening inside so as we see was started and everything actually looks good waiting for connections on Port 27,7 okay so now let's start Express we want Express to connect to the running mongod DB be container on Startup and here we have an example of how to run it and here we have the list of environmental variables that we can configure so let's quickly look at them username password we don't need them however we need the admin username and admin password of the motb this is actually what we overwrote with admin and password so we're going to use them because Express will need some username password to authenticate with the mongodb and to connect it the port is by default the correct one so we don't need to change that um and this is an important part this is the mongodb server right so basically this is the container name that Express will use to connect to the docker and because they running in the same network only because of that this configuration will work if I didn't if I hadn't specify the network then I could have I could specify the name correct name here of the container but it wouldn't work so with that said let's actually create the docker run command for Express as well so let's clear the history and let's start so again we run it in detached mode and let's see what parameters we need so first of all Port let's say what is the default Port that the express runs on that's 80 81 so we'll take that so basically it's going to run on our laptop on Port 8081 the next option would be these two and remember environmental variables need to be specified with minus E and this is the username of mongodb admin which is admin because we specified it when we started the mongodb container this is the password let's set this one as as as well don't forget the network minus minus net Network we have the name we can also call it Express and let's see what else we might need here yes this is important one um and our container name let's actually see it again toer PS the one running it's called mongodb that's the container name and this is what we need to specify here so I'm going to write this here and finally the image is called Express so I'm just going to copy this one here and that is it so basically with these commands do Express should be able to connect to the mongod Deb container so let's run it and just to make sure let's log the container and see what's happening there waiting for mongodb welcome to express it looks like it connected successfully um it says here database connected and the Express is available at Port 8081 so let's check the Express out at the Port 881 so actually let's close these tabs we don't need them anymore and here if I say Local Host 881 I should be able to see the manga Express so these are the databases that already exist by default in or which are created on Startup and using the UI we can create our own database as we saw previously we could have specified an environmental variable init DB on mongodb Startup and that would have created a new database however it doesn't matter we will just create a database name here so we will call it user um account database so let's create one and now we can actually use it or connect to this database from node.js so let's see how that works so now we have the mongodb container and the Express container running so let's check that we have both of them we'll have to connect nodejs with the database so the way to do it is usually to give a protocol of the database and the URI and the URI for a mongodb database would be Local Host and the port that it's accessible at I already went ahead and prepared the code for node.js so basically we are going to use a client here which is a node module and using that client we are connecting to the mongodb database so this is the protocol the host and the port that we just saw that the mongodb is listening at and username and password of the root user of mongodb of course usually you wouldn't put the password here or not use an admin or root uh username password to connect to a database but this is for just the demonstration purposes and these are username and password that we set as environmental variables when we created the docker mongodb container so let's check that so this is the mongodb uh container command and this is the username root and root password that we specified and this is what we are going to use in the code as I said for demonstration purposes I will write the password directly here so then we connect to the database um so I also went ahead and in the Express user account database and inside that I created collection which is like a table in my SQL world called users so here I connect to user account database and I query The Collection users and this is a get request so I'm just fetching something from the database and this is update uh request same thing I connect to the database using the same URI and the database name and I update or insert something in the collection so now let's see how all that works so let's head over to the UI so in the users collection there is no data it's empty so we're going to refresh it and edit the data so I'm going to write here some and update and refresh we see that a new insert was made so this is the update profile section here so all this was executed it connected to the mongodb and now we have one entry which is email coding name that we changed so if I'm going to refresh it now I fetched the newly inserted user data in the UI and I displayed it here and also if you want to see what the mongod container actually logs during this process we can actually look at the logs so I'm going to say docks and log using the container ID so let's say if I wanted to see just the last part of it because I want to see what the last activity was I can also let's clear this and I can also do tail so I can just display the the last part of it or if I want it I could also stream the logs so I'll clear this again and I'll say stream the logs so I want have to do dockal logs all the time so if I make a line here for example to Mark the last logs I can refresh it let's make some other changes let's change it to own and save profile so I'm going to see some activity here as well so these connections are new and it also says received client metadata and this is where the nodejs request comes in with the nodejs and its version and at the end of each communication there is an end connection because we end the database connection at the end so we see that also in the logs so if for example something wasn't working properly you could always check them in the logs here so with that I have a fully functional JavaScript nodejs application which has a persistence in the mongodb database and we also have uh UI both of them running in a Docker container so this would be uh somehow an realistic example of how local development using Docker containers would look like so in the last video we created and started two Docker containers mongodb and Mong Express and these are the commands that we used to make it happen right the first we created a network where these two containers can can talk to each other using just the container name and no host Port Etc is necessary for that um and then we actually ran two Docker run commands with all the options and environmental variables Etc set now uh this way of starting containers all the time is a little bit tedious and you don't want to execute these run commands all the time on the command line terminal especially if you have a bunch of Docker containers to run you probably want to automate it or just make it a little bit easier and there's a tool that's that makes running multiple Docker containers with all this configuration much easier than with Docker run commands and that is Docker compose if you already know Docker comp post and you are wondering why is it useful and what it actually does then bear with me in the next slide I'm going to explain that so this is a Docker run command of the mongodb that we executed previously so basically with Docker compose file what we can do is we can take the whole command with its configuration and map it into a file so that we have a structured commands so if you have have let's say 10 Docker containers that you want to run for your application and they all need to talk to each other and interact with each other you can basically write all the Run commands for each container in a structured way in the docker compos and we'll see how that structure actually looks like so on the right side in the docker compos example the first two tags are always there right version three that's the latest version of the compose Docker compose and then we have the services this is where the container list goes so the first one is mongodb and that Maps actually to The Container name right this is going to be a part of container name when Docker creates a container out of this configuration blueprint the next one is actually the image right so we need to know which image that container is going to be built from and of course you can specify a version tag here um next to the name the next one one is Port so we can also specify which ports is going to open first one is on the host and the second one after the colum is on the container so the port mapping is there and of course the environmental variables can be also mapped in the docker compose and this is how actually the structure of Docker compose looks like for one specific commands let's actually see the second container command for Express that we executed it and how to map that so now again we have a Docker run command for Express and let's see how we can map it into a Ducker compose so as I said services will list the containers that we want to create and and and again names Express will map map to The Container name the next one will be the image again you can add a tag here if you want to be um have a specific one then you have the ports 80 to to to 8080 and then you have all the environmental variables again under the attribute environment and this is how the docker compose will look like so basically Docker compos is just a structured way to contain very normal common Docker commands and of course it's it's going to be easier for you to edit the the file uh if you want to change some variables s if you want to change the ports or if you want to add some new options to the Run command so to say and maybe you already noticed the network configuration is not there in the docker compost so this Network that we created we don't have to do it in a Docker compose we go to the next slide because we have the same concept here we have containers that will talk to each other using just the container name so what docker compose will do is actually take care of creating a common Network for these containers so we don't have to create the network and specify in which network these containers will run in and we're going to see that in action right away so let's actually create a Docker compost file so I'm going to paste all my contents here and this is exactly what we saw on the slide and I'm going to save it as a yl and we see the highlighting as well be very aware of the indentation they have to be correct so this is the list of all the containers on the same level and then each container has its configuration inside that so now compared to Docker run commands it will be very easy for me to go here and change these environment variables or add some new configuration options Etc so here again for demonstration we actually save the docker compose in the code so it's part of the application code so now that we have a Docker compose file the question is how do I use it or how do I start the containers using that so let's go to the command line and start Docker containers using this Docker compose file so the way to use it is using Docker compose command now if you've installed Docker on your laptop it usually gets installed with the docker compose packaged inside so you should have both Docker and Docker compose commands installed as a package so Docker compos command takes an argument which is the file so I'm going to specify which file I want to execute and in my case it's called yl and and at the end I want to say what I want to do with this file in this case the command is up which will start all the containers which are in the yl so let's actually check before that there there are no containers running so I don't have anything running here and I'm going to start those two containers okay so there are couple of interesting things here in this output so let's scroll all the way up so we've talked about Docker Network and how we created our own network at the beginning to run the containers inside and I said the docker compos takes care of it and here we see the output where it actually created a network called my app default this is the name of the network and it's going to run those two containers these are actually the names of the containers the do compos created this is what we specified and it just added prefix and suffix to it and it created those two containers uh in that Network so if I actually go here and do Docker Network LS I see the my app default is here so that's one important thing another one is that logs of both containers actually mixed because we are starting both at the the same time as you see the Express has to wait for mongod DB to start because it needs to establish a connection so we here see the locks so mongodb is starting we still get connection refus because it's not started uh completely and somewhere here when mongodb is started and listening for connections Express is able to connect to it so this is something that you can also do with Docker compose uh when you have two containers that where one depends on another one starting you can actually configure this waiting logic in the docker compos okay so now let's see actually that the docker containers are running so we have both of them here you see the container names that Docker compos gave them and one thing here to note is that the Express actually started on Port 8081 inside the container so we can see that here so we are opening a port 8080 on my laptop that actually forwards the request to container at port 8080 one just so that you don't get confused because it was 8080 on the slides so now that we have restarted the containers let's actually check the first one which is Express so it's running on 8080 in the previous example we created a database and the collection which is gone because we restarted the container this is actually another very important concept of containers to understand when you restart a container everything that you configured in that container's application is gone so data is lost so to say there is no data persistence in the containers itself of course that is very inconvenient you want to have some persistence especially when you're working with a database and there is a concept we're going to learn later called volumes uh that makes it possible to have persistency between the container restarts okay so let's actually create the database again because we need it and inside the database we had actually users collection let's create that one as well and that is empty now let's actually start our application and there you go so now if I were to modify this one here and update I should see the updated entry here so the connectivity with mongodb works so now what do I do if I want to stop those containers of course I could go there and say Docker stop and I can provide all the IDS as we did previously or with Docker compose it's actually easier I can do Docker compose again specify the file and instead of up I'm going to say down and that will go through all the containers and shut them all and in addition to remove removing the containers or stopping them removing the containers it also removes the network so the next time we restarted it's going to recreate so let's actually check that the network LS that default my app default Network case G and when I do up see it gets recreated that should give you a good idea of what dock compose is and how to use it the next we're going to build our own Docker image from our node.js JavaScript application so now let's consider the following scenario you have developed an application feature you have tested it and now you're ready to deploy it right to deploy it your application should be packaged into its own Docker container so this means that we are going to build an Docker image from our JavaScript no JS backend application and prepare it to be deployed on some environment to review this diagram that we saw at the beginning of the tutorial so we have developed a JavaScript application we have used the mongodb docker container to use it and now it's time to commit it to the git right so in this case we're going to simulate this steps on the local environment but still I'm going to show you how these steps actually work so after commit you have a continuous integration that runs so the question is what does actually Jenkins do with this application when it builds the application so the JavaScript application using the npm build Etc it packages it then in a Docker image and then pushes it into a Docker repository so we're going to actually simulate what Jenkins does with their application and how it actually packages it into a Docker image on the local environment so I'm going to do all this on my laptop but it's basically the same thing that Jenkins will do and then on later step we can actually push the built image into a Docker repository in order to build a Docker image from an application we basically have to copy the contents of that application into the docker file it could be an artifact that we built in our case we just have three files so we're going to copy them directly in the image and we're going to configure it and in order to do that we're going to use a blueprint for building images which is called a Docker file so let's actually see what is a Docker file and how it actually looks like so as I mentioned Docker file is a blueprint for creating Docker images a syntax of a Docker file is super simple so the first line of every Docker file is from image so whatever image you building you always want to base it on another image in our case we have a JavaScript application with no JS backend so we are going to need node inside of our container so that it can run our node application instead of basing it on a Linux Alpine or some other lower level image because then we would have to install node ourselves on it so we are taking a ready node image and in order to see that let's actually go to dockerhub and search node here and here you see there is a ready node image that we can base our own image from so here we have a lot of different text so we can actually use one specific one or we can just go with the latest if we don't specify any take so what that actually means basing our own image on a node image is that we're going to have node installed inside of our image so when we start a container and we actually get a terminal of the container we can see that node command is available because there's node installed there this is what from node actually gives us so the next one is we can configure environmental variables inside our Docker file now as you know we have already done this in the using the docker on commands or the docker compos so this will be just an alternative to defining environmental variables in a Docker compos for example I would say it's better to define the environmental variables externally in a Docker compos file because if something changes you can actually overwrite it you can change the docker compos file and override it instead of rebuilding the image but this is an option so this n command basically would translate to setting the environment of variables inside of the image environment the next one is run so all these Capital case words that you see from in and run they're basically part of a syntax of a Docker file so using run basically you can execute any kind of Linux commands so you see make directory is a Linux command that creat creates home slome app um directory very important to note here this directory is going to leave inside of a container so when I start a container from this image the slh home/ app directory will be created inside of the container and not on my laptop not on the host so all these commands that you have in Docker file will apply to The Container environment none of them will be affecting my host environment or my laptop environment so with run basically you can execute any Linux commands that you want so that's probably one of the most used ones and we also have a copy command now you would probably ask I can execute a copy command a Linux copy command using run yes you could but the difference here is that as I said all these commands in run for example they apply to they get executed in inside of the container the copy command that you see here it actually uh executes on the host and you see the first parameter is Dot and the second parameter is slome app so source and the Target so I can copy files that I have on my host inside of that container image because if I were to execute run CP Source destination that that command would execute inside of the docker container but I have the files that I want to copy on my host in the last one so from and CMD or command is always part of Docker file what command does is basically executes an entrypoint Linux command so this line with the command actually translates to node server JS so remember here we actually do node server JS so we start a node server with the nodejs this is exactly what it does but inside of the container so once we copy our server JS and other files inside of a container we can then execute node server.js and we are able to do it because we are basing on the Node image that already has node pre-installed and we are going to see all this inaction so another question here what is the difference between run and CMD because I could also say run node server.js the difference again is that CMD is an entry point command so you can have multiple run commands with different Linux commands but CMD is just one and that marks for Docker file that this is the command that you want to execute as an entry point so that basically runs the server and that's it so now let's actually create the docker file and just like the docker compos file Docker file is part of the application code so I'm going to create a new file here and I'm going to paste here the contents so again we're basing off Note image and actually instead of just having the latest node I'm going to specify a node version so I'm going to take 13 minus Alpine so all these that you see here are Texs so I can use any of them as a TCH so I'm going to say 13 minus Alpine like this so this is going to be a specific node image that I'm going to use as my base image let's actually stop here for a moment and take a little bit of a deep dive on this line so since we saw that Docker file is a blueprint for any Docker image that should actually mean that every docker ER image that there is on dockerhub should be built on its own Docker file right so if we actually go to Let's actually look at one of the latest versions which is 13 minus Alpine and let's click inside and as you see this specific image has its own Docker file and here as you see we have the same from that we just saw and this is what this node official image is based of which is a base image Alpine 3.10 right and then we have this environmental variable set and all these Linux commands using run and some other environmental variable and you have this entry point which is a script so you can also execute the whole shell script instead of separate commands and you have this final Command right so you don't have to understand any of this I just want to demonstrate that every image is based of another base image right so in order to actually visually comprehend how this layer stacking works with images let's consider this simplified visualization so our own image that we're building app with a version 1.0 is going to be based on a node image with a specific version that's why we're going to specify from node 13 Alpine and the node 13 Alpine image as we saw in the docker file is based on Alpine base image with a version 3.10 that's why it specifies from Alpine 3.10 so Alpine is a lightweight base image then we install node on top of it and then we install our own application on top of it and basically this is how all the images are built so now let's go back and complete our Docker file so we have the from specified we have the environmental variables specified and in just a second we're going to actually see these commands in action so let's copy that and this is also very important Docker file has to be called exactly like that you can just give it any name it is always called Docker file starting with a capital D and that's it it's a simple text file so just save it like this and here you even see the highlighting and this Docker icon so now that we have a Docker file ready let's see how to actually use it so how do we build an image out of it so in order to build an image using the docker file we have to provide two parameters one is we want to give our image a name in the tag just like all the other images have so we are going to do it using minus t so we are going to call our image my app and we're going to give it a tag of 1.0 the tag could be anything you can even call it actually version one it wouldn't matter so we're going to do 1 point0 and the second required parameter actually is a location of a Docker file because we want to tell Docker here build an image using this Docker file and in this case because we're in the same fold as the docker file we're just going to say current directory when we execute this we're going to see that image is built and this is an ID of the image that was built because I already have node 13 Alpine on my laptop this just use the the one I have lying around locally for you if it's the first time you will actually see that it's pulling node image from the dockerhub so now with the docker images I can actually see that my image is here it says created two days ago I don't know why but anyways so I have the image name which is this one here and I have the name of the image and the tag of the image so if we go back to this diagram that we saw in the review so basically we've got all these steps or we have simulated some of the steps we've built the JavaScript application using a Docker containers and one once the the application is ready let's say we made the commit and we we just simulated what Jenkins server also does so what Jenkins does is actually it takes the docker file that we create so we have to commit the docker file into the repository with the code and Jenkins will then build a Docker image based on the docker file and what is an important Point here is that usually you don't develop long you are in the team so other people might want to have access to that upto-date image of your application that you developed it could be a tester maybe who wants to pull that image and test it locally or you want that image to be deployed on a development server right and in order to do that you have to actually share the image so it is pushed into a Docker repository and from there either people can take it for example a tester maybe want to download the image from there and test it locally or a development server can actually pull it from there so let's actually just run a container I'm just going to say Docker run the image name obviously and a tag like this and in this case I'm not going to specify any other options because we just want to see what's going on inside of the container so I'm just going to run it okay so the problem is that it can't find the server JS file which is actually logical because we are not telling it to look in the correct directory so since we are copying all the resources in this home/ home/ app directory server JS is going to be there as well and this is another topic whenever you adjust a Docker file you have to rebuild an image because the old image cannot be over written so to say so what I'm going to do now is actually I'm going to delete the one that I built so I'm going to I'm going to actually take the image this is how you delete an image but I can delete it because as as it says the docker is used by a stopped container so if I do Docker PS minus a actually let's crap to my app like this I have to first delete the container so this is how you delete a container it's doer RM and once I've deleted the container I can delete an image so the image deletion is RMI like this so if I do images now I see my image isn't there okay so we' have modified the docker file so let's rebuild it now so Docker build again and let's see the image is here so let's start it again so it's my app 1.0 and let's run it and as you see the problem is fixed app listening on Port 3000 so our app is running so this one here I app 1.0 first of all we can see the logs here like this we see that the EP is listening on Port 3000 we know everything is cool to actually just get a little bit more inside let's enter the containers or let's get the terminal the command line terminal of the container and look around there so I'm going to say Docker exit interactive terminal I'm going to specify the container ID ID ID in like this and since bin bash doesn't work we can actually try shell so this is something you will also encounter because some containers do not have bash installed so we'll have to connect using bin sh so one of them has to work always so let's see in which directory we are so we are in the root directory and we see our virtual file system there and as you see the cursor changed as well so that means we're inside of a container so now let's actually check some of the stuff so first of all we specified some environmental variables here in the docker file and this means that these environmental variables have to be set inside the docker environment so if we do inv we actually see the mongodb username this one here and mongodb password are set and there are some other environmental variables automatically said we don't care about them so another thing we can check is this directory because remember because with this line we actually created this slome app directory so let's see slome Slash app and as you can see the directory was created and with the next Land We copied everything in the current folder so if we actually go and see reveal in finder so this is where the docker file resides so basically we copied everything that is inside of this directory so all of these into the Container now we don't actually need to have Docker file and Docker compose and uh this other stuff in here because the only thing we need are the JavaScript files or if we build a JavaScript application artifact just the artifact so let's go ahead and improve that so what I'm going to do is I'm going to create an app directory and I'm going to copy just the files that I'm going to need for starting an application inside of a container so I'm going to take those and the images as well so all these are just external ones we don't need them there and images the index HTML file package Jon server JS and node modules are inside of app so what we can do it now is instead of copying the whole directory where where the docker file is I just want to copy all the contents of EP folder so what I'm going to do is I'm going to say copy all the app app app contents and again because we modified a Docker file we need to recreate the image in order to leave the docker Container Terminal can actually exit so now we are on the host again so if I do Docker images again I have to first delete the container and then image but in order to delete the container I have to first stop it so now I can remove the container and now I can actually remove the the the image that the container was based on and let's check again so let's ex execute that build command again so now that we have the image built let's actually run it so I'm going to say my app app app 1.0 and of course I could have executed with a minus D in a detached mode it doesn't matter now and if I do a Docker PS I see my um image container running and now let's actually enter the container G so it and as we learned it was in sh and again we're going to see the home app and here we just have the contents of app directory so no unnecessary Docker file Docker compose Etc files which is actually how it's supposed to be or as I said because I just had a couple of files here I copied all of them but usually if you have this huge application you would want to compress them and package them into an artifact and then copy that artifact into a Docker image container okay but as I said this was just for demonstration purposes because I just wanted to show you um how you can actually start it as a container and how it should to look inside and in this case we improved a couple of things but usually we would start this container from a Docker compose as well together with all the other Docker images that the application uses and it's also doesn't have any ports open so uh this is just for demonstration purposes so in this video we're going to create a private repository for Docker images on AWS ECR there are many more options for Docker Registries among them Nexus and digital ocean so we're going to see how to create a registry there build and tag an image so that we can push them into that repository and in order to push the images into a private repository you first have to log into that repository so let's see how it all works so the first step is to actually create a private repository for Docker it's also called Docker registry in this case we're going to do it on AWS so let's see so I already have an account on AWS so the service that we're going to use is called elastic container registry so so so ECR doer container registry and because I don't have a repository there yet I am presenting with the screen so in order to create a repository click on get started and here we have a repository name and we're actually going to name it the name of the application that we have so I'm actually going to name it my app this is the domain of the registry from AWS and this is the repository name which is the same as my image name and don't worry about the other stuff right now and just create a repository it's as simple as that now one thing I think specific to Amazon container service is that here you create a Docker repository per image so you don't have a repository where you have where you can actually push multiple images of different applications but rather for each image you have its own repository and you go inside of the repository here it's empty now but what you store in a repository are the different tags or different versions of the same image so this is how the Amazon container service actually works there are other dep Docker Registries that work differently for example you create a repository and you can just throw all of your container images inside of that one repository so I think this is more or less specific for AWS so anyways we have repository which is called my app and let's actually see how we can push the image that we have locally so actually check that once more so we want to push this image here into that repository so how do we do that if you click on this one the view push commands will be highlighted this is different for each registry but basically what you need to do in order to push an image into repository are two things one you have to log in into the private repository because you have to authenticate yourself so if you are pushing from your local laptop or local environment you have to tell that private reposit hey I have access to it this is my credentials if a Docker image is built and pushed from a Jenkins server then you have to give Jenkins credentials to login into the repository so Docker login is always the first step that you need to do so here AWS actually provides a Docker login command for AWS so it doesn't say Docker login but in the background it uses one so I'm going to execute this login command for AWS Docker repository uh so in the background it uses actually Docker login to authenticate so in order to be able to execute this you need to have AWS command line interface and the credentials configured for it so if you don't I'm going to put a link to the guide of how to do that in the description I have configured both of them so I can execute this command and I should be logged in successfully to the docker repository so now I have authenticated myself to the docker repository here so I'm able to push the image that I have locally to that repository but before I do that there is one step I need to do so I've already built my image so that's fine and now I have to tag my image and if this command here looks a little bit too complicated for you or too strange let's actually go and look at image naming Concepts in Docker repositories so this is the naming in Docker Registries this is how it works the first part of the image name the image full name is the registry domain so that is the host Port Etc slash repository or image name and the tag now you may be wondering every time we were pulling an image out of dockerhub we actually never had this complex long name of the image right so when we were pulling an image it looked like this Docker pole 4.2 the thing is with dockerhub we're actually able to pull an image with a short hand without having to specify a registry domain but this command here is actually a shorthand for this command what actually gets executed in the background when we say Docker pole is Docker pole the registry domain so docker.io library is a registry domain then you have the image name and then you have the tag so because we we were working with dockerhub we were able to use a shortcut so to say in a private Registries we can just skip that part because there is no default configuration for it so in our case in AWS ECR what we going to do is we're going to execute Docker pull the full registry domain of the repository this is what we're going to see here and a tag and this is how AWS just generates the docker registry name that's why we see this long image name with the tag here and we have to tag our image like this so let's go back and take a look at our images our image that we built again and under the repository it says my app now the problem is we can just push an image with this name because when we say Docker push my app like this Docker wouldn't know to which repository we're trying to push by default it will actually assume we're trying to push to dockerhub but it's not going to work obviously because we want to push it to AWS so in order to tell Docker you know what I want this image to be pushed to AWS repository with the name my app we have to tag the image so we have to include that information in the name of the image and that is why we have to tag the image tag basically means that we are renaming our image to include the repository domain or address and the name okay and AWS already gives us the command that we can execute we want to use the specific version so I'm going to use 1.0 in both so what this is going to do is it's going to rename this is what tech does my app 1.0 this is what we have locally this is what the name is to this one here so let's execute that and let's see what the outcome is and as you see it took the image that we had it made a copy and renamed it into this one so these two are identical images they're just called in a different way and now when we go back we see the docker push command so basically this thing here is a same as Docker push and name of the image and the take so this push command will tell Docker you know what I want you to take the image with tag 1.0 and push it into a repository at this address so when I execute this command see the push command will actually push those layers of the docker image one by one this is the same thing as when we're pulling it we also pulled the images layer by layer and this is what happens in the reverse Direction when we push it so this is also going to take a little bit great so the push command was complete and we should be able to see that image in the AWS repository now so if I go inside see I have image tag with 1.0 this is our tag here and push the time the digest which is the unique hash of that image and the image URI which is again the name of the image using the the repository address image name or repository name in this case and the tag so now let's say I made some changes in the docker file you know let's say I renamed this home slome to node app like this or what could also lead to need to recreate an image is obviously when I change something in the code right so you know let's say I were to delete this line because I don't want to console log to be in my code and now I have a different version of the application where I have changes in the application so now I want to have those changes in the new Docker image so now let's build a new Docker image out of of of it so Docker build let's call it my app with a version 1.1 and a path to a Docker file and now I have a second image which is called my app with version 1.1 so now again because I want to push this to repository I have to rename it to include the repository address inside of it so I'm going to do Docker tag the first parameter is the image that I want to rename and the second one is the name of that image a new name so it's going to be the same as the previous one because the repository name and the address is the same remember we have one repository for one image but for different versions so we're building a version 1.1 so it should end up in the same repository so now here we have 1.1 and if I tag that and images I have a second image here so I'm going to copy that and I'm going to do Docker build and do not forget that tag it's important because because it's part of the complete name sorry it's Docker push and now some of the layers that I already pushed are there only the ones that changed are being rep pushed sort of say and also know that I just have to do Docker login once at the beginning and then I can pull and push images uh from this repository as many times as I want so do login is done once so now that is complete let's actually reload this so my repository now has two versions so this is pretty practical if you are for example testing with different versions and you want to have a history of those image Texs if you want to for example test a previous version and I think in AWS the repos each repository has a capacity of hold holding up to 1,000 uh image versions so for example my app here can have thousand different tags or of the same image okay so now going to compare it to the initial diagram that we saw for this complete flow let's actually switch back to it quickly so here what we did is basically simulate how Jenkins would push an image to a Docker repository so whatever we did on our lap top will be the same commands executed on a Docker on the Jenkins server and again Jenkins user or Jenkins server user has to have credentials to the docker repository to execute Docker login depending on the registry or repository configuration will look different and Jenkins needs to tag the image and then push it to the repository and this is how it it's done and the next step of course we need to use that image that is lying now in the repository and we're going to see how it's pulled from that repository and again we're going to do it on the local environment but it's the same thing that a development server or any other environment will actually execute so in this video we're going to see how to deploy an application that we built into a Docker image so after you package your application in a Docker image and save it in the private repository you need to somehow deploy it on a development server or integration server or whatever other environment and we're going to use Docker compose to deploy that application so let's imagine we have logged in to a development server and we want to run our image that we just push the repository so our my app image and the mongodb image uh both the database and the Express on the development server so the my app image will be pulled from private repository of AWS the in the two containers will be pulled from the docker Hub so let's see actually how that would work so usually again you have developed your application you done with it and you have created uh your own Docker image right now in order to start an application on development server you would need all the containers that make up that application environment okay so we have mongodb and Express already so what we are going to do is here we're going to add a new container in the list which is going to be our own image so let's go ahead and copy the image from our repository so let's actually use the 1.0 so again remember we said that this image name is a shortcut for having a docker.io do library SL with like a specific version so instead of that because we are pulling these images from a Docker Hub we can actually skip that repository domain in front of the images but here because we're pulling it from a private repository so if we were to specify our image like this Docker will think that our image resides on dockerhub so we try to pull it from dockerhub and of course it won't find it because we have to tell Docker go and look at this repository with this repository name and this TCH and of course in order to be able to pull this image or the docker composed to be able to pull this image the environment where you execute this Docker compost file has to be logged into a Docker repository so here as the development server has to pull the image from the repository what we would need to do on the development server is actually do a dock login before we execute the docker compose and obviously you don't need a Docker login for Docker Hub those images will be pulled freely okay so the next thing that we have to configure are the ports because obviously want to open the ports if we go back we see that our application runs on Port 3000 so the port of the container or the where the container is listening on is 3,000 and here we can open the port on the host machine so it's going to be 3,000 me to 3,000 we have actually the environment variables inside of the docker file but obviously we could have configured them in the docker compose just like this so it's an alternative so this will be a complete Docker compost file that will be used on the development server to deploy all the all the applications inside so again if we're trying to simulate a development server the first step will be to do the docker login in this case you have this on command for logging into the AWS Repository which I have done already in this terminal so the next step is to have the docker compos file available on this development server because we have to execute the docker compost file because we're simulating here the way I would do it is I'm going to create a yl file in the current directory where I am I'm going to copy this and save so now I have my ml file and now we can start all three containers using Docker compose command minus f up and here we see that app started on 3000 and mongodb and express started as well so let's check again now and here we saw that database is lost every time we recreate a container and of course that's not good and we're going to learn how to preserve the database data between the container restarts using Docker volumes in the later tutorials because this is not an ideal State okay so now that we have database in a collection let's actually refresh and our application works as well let's check awesome so application works let's refresh this one as well and there is actually one thing that I needed to change in the code to connect nodejs with mongodb so let's actually go and look at that these are my handlers you know nodejs where I connect to the mongodb database so the uis are the same and what I changed here is that it was a local host before so instead of Local Host I changed it to mongodb because this actually is a name of the container or of the service that we specify here so this actually leads back to the docker Network and how Docker compos takes care of it is that in the URI or when I connect one application in a Docker container with another one in another Docker container I don't have to use this uh Local Host anymore actually I wouldn't even need to use the port even because I have all that information so the host name and the port number in that configuration so my application will be able to connect to mongodb using the service name and because of that you don't have to specify here a local host and a port number which is actually even more Advantage when you consider using Docker containers to run all of your applic ations because it makes the connectivity between them even more easier and that actually concludes the this uh diagram that we saw previously we have gone through all of the steps where we saw uh how to develop uh a JavaScript application locally with Docker containers then we saw how to build them into an image uh just like a continuous integration build will do it then we push it into a private repository and we simulated a development server where we pulled the image from U private repository and the other images from the dockerhub where we started the whole application setup with our uh own application and the two applications uh using a Docker compose which is how you would deploy an application on a Dev server so that now testers or other developers will be able to um access the development server and actually try out the applic that you just deployed or you can also use it for demos so in this video we're going to learn about Docker volumes in a nutal Docker volumes are used for data persistence in Docker so for example if you have databases or other stateful applications you would want to use Docker volumes for that so what are the specific use cases when you need Docker volumes so a container runs on a host let's say we have a database container and a container has a virtual file system where the data is usually stored but here there is no persistence so if I were to remove the container or stop it and restart the container then the data in this virtual file system is gone and it starts from a fresh state which is obviously not very practical because I want to save the changes that my application is making in the database and that's where I need Docker volumes so what are the docker volumes exactly so on a host we have a physical file system right and the way volumes work is that we plug the physical file system path it could be a folder a directory and we plug it into the containers file system path so in simple terms a directory a folder on a host file system is mounted in into a directory or folder in the virtual file system of Docker so what happens is that when a container writes to its file system it gets replicated or automatically written on the host file system directory and vice versa so if I were to change something on the host file system it automatically appears in the container as well so that's why when a container restarts even if it starts from a fresh state in its own virtual file system it gets the data automatically from the from the host because the data is still there and that's how data is populated on a startup of a container every time you restart now there are different types of Docker volumes and so different ways of creating them usually the way to create Docker volumes is using Docker run command so in the docker run there is an option called minus V and this is where we Define the connection or the reference between the host directory and the container directory and this type of volume definition is called host volume and the main characteristic of this one is that you decide where on the host file system that reference is made so which folder on the host file system you mount into the Container so the second type is where you create a volume just by referencing the container directory so you don't specify which uh directory on the host should be mounted but that's taking care of the docker itself so that directory is first of all automatically created by Docker under the VAR leap Docker volumes so for each container there will be a folder generated that gets mounted automatically to the container and this type of volumes are called Anonymous volumes because you don't have a reference to this automatically generated folder basically you just have to know the path and the third volume type is actually an improvement of the anonymous volumes and it specifies the name of the folder on the host file system and the name is up to you it's just to reference the directory and that type of volumes are called named volumes so in this case compared to Anonymous volumes you H you can actually reference that volume just by name so you don't have to know exactly the path so from these three types the mostly used one and the one that you should be using using in production is actually the named volumes because there are additional benefits to letting Docker actually manage those uh volume directories on the host now they showed how to create Docker volumes using Docker run commands but if you're using Docker compose it's actually the same so this actually shows how to use volume definitions in a Docker compose and this is pretty much the same as in Docker run commands so we have volumes attribute and underneath you define your volume definition just like you would in this minus V option and here we use a named volume so db- data will be the name reference name that you can just think of could be anything and in vly MySQL data is the path in the container then you may have some other containers and at the end so on the same level as the services you would actually list all the volumes that you have defined you def find a list of volumes that you want to mount into the containers so if you were to create volumes for different containers you would list them all here and on the container level then you actually Define under which path that specific volume can be mounted and the benefit of that is that you can actually mount a reference of the same uh folder on a host to more than one containers and that would be beneficial if those containers need to share the data in this case you would Mount the same volume name or reference to two different containers and you can mount them into different path inside of the container even in this video we are going to look at Docker volumes in practice and this is a simple nodejs mongodb application uh that we're going to attach the volume to so that we don't lose the database data every time we restart start the mongodb container so let's head over to the console and I'm going to start the mongodb with the docker compose so this is how the docker compose looks like we're going to start the mongod TB uh container and the Express container so that we have a UI to it so I'm going to execute the docker compost which is going to start mongodb and the Express so when it started I'm going to check that Express is running on port 8080 and here we see just the default databases so these are just created by default on Startup um and we're going to create our own one for the node.js application and inside of that database I'm going to create users collection so these are the prerequisites or these are the things that my node.js applic ation needs so this one here in order to connect to the database might DB this is what we just created ITB and inside of that to the collection called users so let's start the application which is running on Port 3000 so here and this is our app which when I edit something here we'll write the changes to my database now if I were to restart now the mongodb container I would lose all this data so because of that we're going to use named volumes inside of the docker compos file to persist all this data in the mongodb let's head over to dock compose so the first step is to Define what volumes I'm going to be using in any of my containers and I'm going to do that on the services level so here I Define the list of all the volumes that I'm going to need in any of my containers and since we need data persistency for mongod TB we're going to create uh data volume here now this is going to be the name of the volume reference uh but we also need to provide here a driver local so the actual store storage path that we're going to see later once it's created it's is actually created by Docker itself and this is kind of an information additional information for Docker to create that physical storage on a local file system so once we have a name reference to the volume defined we can actually use it in the container so here I'm going to say volumes and here I will Define a mapping between the data volume that we have on our host and the second one will be the path inside of the mongodb container it has to be the path where mongodb explicitly persists its data so for example if you check it out online you see that the default path where mongodb stores its data is data/ dat DB and we can actually check that out so if I say docker s and go inside the container it's minus it I can actually see data DB and here is all the data that mongodb actually holds but this is of course only the container so when the container restarts the data get regenerated so nothing persists here so this is the path inside of the container not on my host that we need to reference in the volumes here so we're attaching our volume on the host to to to data/ dat DB inside of a container so for example for MySQL it's going to be um VAR leap MySQL for postgress it's also going to be VAR leap postris SQL SL data so each database will have its own so you have to actually find the right one so what this means is that all the data with that we just saw here all of this will be replicated on a container startup on our host on this persistent volume that we defined here and vice versa meaning when a container restarts all the data that is here will be replicated inside of that directory inside of a container so now that we have defined that let's actually restart the docum compose and restart it so once we create the data and I'm going to the collection and let's actually change this one one one here and update it so we have a data here so now that we have the persistent volume defined if I were to restart all these containers these data should be persisted so in the next restart I should see the database my DB collection and the entry here so let's do that I'm going to do down great so let's check see the database is here the collection is here and the entry has persisted so now let's actually see where the docker volumes are located on our local machine and that actually differs between the operating systems for example on a Windows laptop or a computer uh the path of the docker volume will be at program data Docker SL volumes the program data Docker folder actually contains all the other container information so you would see other folders in this Docker directory besides the volumes on Linux the path is actually /ar leap Docker volumes which is comparable to the windows path so this is where the docker saves all this configuration and the data and on the mech it's also the same one inside of this volumes directory you actually have a list of all the volumes that one or many containers are using and each volume has its own hash which is or which has to be unique and then slore data will actually contain all the files and all the data that is uh persisted let's head over to the command line and actually see um the volumes that we persisted for mongodb now interesting note here is that if I were to go to this path that I just showed you in the presentation which is VAR Le Docker see there is no such directory so that could be a little little bit confusing but the way it works on Mac specifically on Linux you would actually have that path directly on your host but on Mech it's a little bit different so basically what happens is that Docker for Mech application seems to uh actually create a Linux VM uh in the background and store all the docker inform or doer data about the containers and the volumes Etc inside of that vm's storage so if we execute this command here so this is actually the physical storage on my laptop that I have where all the data is stored but if I execute this command I actually get the terminal of that VM and inside here if I look I have a virtual different virtual file system and I can find that path that I showed you here so it's VAR leap Docker see so I have all this Docker information here I have the containers folder and I have volumes folder so this is the one we need sort of that actually go to the volumes and this is a list of volumes that um I have created and this is the one that came from our Docker compose right this is the name of our app this is do this is what Docker compose actually takes as the name we can actually take a look here so when it's creating these containers it depends these name as a prefix and then there is mongodb and our volume has the same pattern it has the prefix and then mongod data this is the name that we defined here so now if we look inside of that mongod data volume directory we see that underscore data this would be the anonymous volumes so basically here you don't have a name reference it's just some random uh unique ID but it's the same kind of directory as this one here the difference being that this one has a name so it's more it's easier to reference it with a name so this is an onymous volume this is a named volume but the content will be used in the same way um so here as you see in this underscore data we have all the data that mongodb uses so this will be where it gets the default databases and the changes that we make through our application inside and if I go inside of the container so remember this volume is attached to to to mongodb and is replicated inside of the container under path SL dat DB so if we go go inside of the container actually it here PS SL DP we'll see actually the same kind of data here so we have all this index and collection um files just like we did in this one so now whenever we make changes to our application for example we change it to SM whatever and this will make the container update its data and that will Cascade into this volumes directory that we have here so that on the next startup of a container when the SL data/ DB is totally empty it will actually populate this directory with the data from this uh persistent volume so that we will see all the data that we uh created through our application again on Startup and that's how do loer volumes work in order to end that screen session that we just started because exit doesn't work in this case uh somehow on Mac you can actually click on control a k and then just type Y and the session will be closed so when you do screen LS you should see actually it's terminating congratulations you made it till the end I hope you learned a lot and got some valuable Knowledge from this course now that you've learned all about containers and Docker technology you can start building complex applications with tens or even hundreds of containers of course these containers would need to be deployed across multiple servers in a distributed way you can imagine what overhead and headache it would be to manually manage those hundreds of containers so as a Next Step you can learn about container orchestration tools and kubernetes in particular which is the most popular tool to automate this task if you want to learn about kubernetes be sure to check out my tutorials on that topic And subscribe to my channel for more content on modern devops tools also if you want to stay connected you can follow me on social media or join the private Facebook group I would love to see you there so thank you for watching and see you in the next video

hi so you're a developer well you think you're a developer nice building apps is pretty cool except when it's not let's say you're building a web app the web app works perfectly hey it's not working what yeah it's not working but it works on my machine this has happened since the beginning of Technology let me show you hey it's not working it works on my machine now this has happened or will happen to every programmer we call this developers law any code that works perfectly on your machine will inevitably fail on everyone else's I made that up by the way but what if I told you there's a way to prevent this this is Docker Docker I hardly know her so what's Docker well it's the sponsor of today's video oh no wonder Docker is the number one most used developer tool based off a stack Overflow survey that helps you develop ship and run applications within lightweight containers what okay so Docker is actually pretty simple and something you should know sloth why should I know Docker you should know Docker because a lot of tech companies use it because it makes development and deployment easier yeah that makes sense plus it's a skill that's in a lot of job posting which means learning Docker will make you money yeah that makes sense what makes Docker so good Docker essentially allows us to package up our code imagine Docker like a Lunchable the Lunchable has your trash code dependencies like no. JS or python environment settings and everything else needed to run your trash code you can bring that Lunchable anywhere and give it to anyone at school at work you can let your friend have it I like my cheese drippy bro no matter where or who uses it it works so basically Docker makes it easier to run your code on any computer because it has everything you need to run the code no need to install anything Docker does it all for you yeah that makes sense now how does Docker do that Docker doeses this with two important Concepts that you have to know images and containers everything revolves around these two concepts so don't forget them you can think of images as the recipe it contains all the ingredients and instructions this means a Docker image contains the technology we need run time and any system tools we need to run the specific code now we need something to actually run the code which is where containers come in a container is like the actual meal it's what gets made from the recipe now the cool part about containers with one image we can create multiple container instances what now what does this mean it means as long as you have the docker image you can create a container and run the code you want proof here's me running a Docker image that contains Doom yeah the entire game of Doom to be fair you can run doom on almost anything but this is what makes Docker amazing it allows us to simplify the process of sharing code you no longer have to do a tedious process like installing all these tools and making these configurations to run some code all you need to do is run one Docker command and it does the rest for you yeah that makes s i simplified this explanation a lot because a lot of you have a shorter attention span than a goldfish so if you're a nerd and you actually want to learn more about Docker I'm going to have links in the description if you want to do your own research or you can check out my free newsletter called sloth bites where I give you free programming information every week to make you a better programmer and did I mention it's now that you understand why Docker is important and how it works kind of let me show you how to actually use it step one download Docker obviously go to doer. or click the link in the description and download Docker desktop it'll download Docker for you and it comes with a nice little gooey to manage your images and containers but if you're feeling like a grownup it also comes with the docker CLI so you can do everything in the terminal step two check if you have Docker open up your terminal and type Docker D- version if you see this you have Docker installed if you don't then uh good luck little bro figure it out step three set up Docker on your IDE of choice I'm doing vs code because I hate myself make sure to download the docker extension on your IDE because it'll have language support and other goodies that will make your life so much easier step four the docker file this is pretty important now what's a Docker file remember our recipe analogy this is literally where we write our recipe let's write one together here we have a simple node server yeah you heard me node JavaScript first we're going to create the docker file and inside this file we're going to write the steps and instructions to run the server now in order to write the steps you need to know the steps so you need to understand how to run the server but since we're using JavaScript it's pretty easy first you get your code then you go into the file or directory download your dependencies and then you run the server with the mpm start command bada bing bada boom easy peasy now that you understand how to run the server all we have to do is write all that in our Docker file what so at the top of your Docker file you're going to write from and if you downloaded that Docker extension I told you to get like the good boy you are if you hover over it you're going to get some nice documentation every Docker file starts with this command now we have to select a base image and a base image is basically just our starting point we can use the officially supported node.js base image that comes with everything we need so in our Docker file we're going to write from node colon 22 all we have to do now is write the instructions to start the server so underneath the fir command you're going to type worker slapp worker stands for working directory and this is basically us telling Docker hey start at this point this is where our code's going to be the next step is a very important step so you better listen to this step usually in a node project we get our code and then download our dependency we could do that in Docker but that's not the best way instead what we'll actually do is we're going to download our dependencies first and then get our code so we're going to write these commands this First Command copies our package files and the second command well it should be pretty obvious it installs our dependencies now why are we doing this unlike you and me Docker is actually pretty smart it has this thing called layer caching you can basically imagine every step that we write as a layer and Docker looks at every layer individually so for layer caching it goes like this layer changed do it again layer did not change use cash that's it so the reason we do this for our dependencies is every time we change our code it's not going to download the dependencies again it's just going to use the cash so it basically skips this step which makes our builds way faster our next step now is to copy the code into the image we do that with this command wow now this command copies everything all our files and our folders into the docker image but this is a problem since we're copying everything that also means we're copying our node modules that's not good because this file is big but not as big as your M we need a way to ignore that file we can do that with a Docker ignore file and inside that file we're going to add the node modules folder and any other files you don't want in there on to the next step in our code we have this port environment variable we need to add this into our darker file we can do that with the EnV command now since this environment variable is a port we're also going going to need this command this basically tells Docker hey we're going to need this port now it's time for the last step running the server we do that with this command whoa whoa whoa whoa why'd you do it like that and why didn't you just use Run mpm start like you did for install good question actually the reason is because the Run keyword happens when we're building the image while the CMD keyword is what Docker uses to actually start the container if we use the Run keyword for this step the container would never start anyways enough Yap our Docker file is complete step five building the image we're going to have to be grown-ups here and use our terminal to build your image we use Docker build command so inside your terminal you're going to type Docker build- t t stands for tag and it basically means give your image a name tag understand so give your Docker image a name you can name it whatever you want the last thing we're going to add is the path to our Docker file which in our case is the current working directory so all we have to do is type A period run this command and you're going to see Docker running all the steps we wrote in our Docker file step six running the container we can do that with the docker run command make sure you write the name you gave your image hey what's that Das P this- P is for port forwarding you can do your own research search for that but for now think of it as a bridge between our computer and our container the number on the left is the port for our computer and the number on the right is the port for the container you don't add this part it's not going to work go ahead try it I dare you run the command and you're going to see that your node server is now running congratulations buddy you know the basics of Docker I'm proud of you step seven debugging even though Docker makes it easier for us to share code it doesn't stop us from building our containers wrong or from writing trash code luckily debugging and checking any logs is pretty easy with Docker desktop so open up Docker desktop top and look at the container that's running it should have a little green dot click the container and you can view any logs you can execute any terminal commands and you can see how the container's doing pretty simple now if you think you're too good for a gooey you can also do this in the terminal with the docker exact command step eight Docker Scout now if you notice when you ran the docker run command Docker gave you a recommendation to run this command Docker Scout quick view now what's Docker Scout it's a tool created by Docker that looks inside your Docker image and it generates a detailed report of all the packages that's inside your image and it's looking for any vulnerabilities inside those packages if it detects any vulnerabilities it'll give you suggestions on how to handle them using Docker Scout is pretty simple you can do it with the terminal or with Docker desktop I personally recommend Docker desktop it's really easy let me show you how to use it all you have to do is go to your images click on the image that's running and once you're there you're going to see this vulnerabilities tab with a button that says start analysis click the button and you're done it's that simple step number nine Docker compose and Docker volumes let's be honest here this note example was really simple nowadays babies in the womb can do this example what if we wanted to add a database what if we wanted to add a front end how do we add that with Docker now you could combine all of them into one giant container but I don't recommend that a lot of people don't recommend that please don't do that it's pretty stupid now the way people usually do this is they separate each of these services and put them in their own containers and this is called a multicontainer application what an obvious name right but this creates a brand new problem now that you have multiple containers you have to run those containers separately and you have to find a way to connect them all together and manage them this this is a lot of work and a lot of areas could go wrong trying to manage these containers lucky for us a lot of programmers are lazy just like me which is why we have this tool called Docker compos Docker compose is pretty easy to understand it basically tells all your containers how to work together to create the full application let me show you a quick example on how to use it let's pretend we added a database to our node server so the first thing we're going to do is create a composed. yo file and inside that file we're going to create an object called services and inside that service object we're going to be writing down keys that represent the containers we want to run so in our case our back end and our database since our backend is using a Docker file we created we're going to use the build key and the value is going to be the location of our Docker file which in our case is the current working directory so it's going to be a period and remember our backend needs port forwarding so we're going to add that in our configuration to and now moving on to our brand new database container what we're going to put inside of this one is the image key because we're not using a Docker file for this database instead what we're going to do is use a base image of the database we want to use which for this example I'll go with postgress now now our database also has some environment variables we have to add so we can do that with the environment key and underneath that we just add all the information okay we're almost done I swear I pinky promise we just need to add one more thing and that's a volume whenever we close our containers we lose all the state and data and our containers right now don't share data between each other so we need to add a volume what is that all right a volume is basically just a folder on our computer that Docker can access to save any data from our containers and to share that data with other containers and it's pretty easy to create inside your Docker compos file you're going to copy down this stuff and this line right here says Hey Docker create a volume called postgress data and the other one tells Docker hey we have this volume here this is how you connect to it our Docker compose file is now complete all we have to do now is run this command in our terminal Docker compose up and Docker will find this configuration file and run all the containers together and if you want to shut down your containers then run this command Docker compose down and that's all you need to know about Docker compose because I'm too lazy to teach you anything step 10 Docker build Cloud if you noticed building our containers takes a bit of time even for the simple notes server now imagine how long these containers take for a complex application apparently based on this 2022 survey from incredib build the average developer loses an hour just waiting on their Docker build Docker realized this problem and recently created something to help reduce build times they created Docker build cloud and the main difference is it builds your containers using the cloud yeah it's it's in the name now when docker's words using the cloud allows your builds to be up to 39 times faster plus if you're working with a team you can share the same cash with them so one person can build the image and now everybody can use that same cash which speeds everything up for everybody so if you have a more complex project or have a Docker project that takes forever to build you should probably check out Docker build Cloud I'm not going to show you how to do it because I don't have a complex project uh step way okay so I'm not going to lie to you all I kind of wasted your time you could have just done Docker and N in your terminal and Docker would have created everything for you so uh thanks for letting me waste your time bye

for the longest time Docker was a tool that I used sparingly throughout my whole career I mean I'm a Ruby on Rails Dev I don't have to know Docker I can just run all my services locally on my machine right I can even remember throughout my career if anyone ever mentioned that we needed to use Docker to run something I would always say Docker barely even know her and as hilarious as that joke is I was missing out on a core foundational piece of technology that I should have known about all along in this video we're going to cover virtualization and containerization and what the difference is between the two things because for me this was always messed up in my head we're also going to cover Docker files images Docker containers and how they all fit into the grand scheme of Docker this video is going to be a gentle introduction to Docker and we're just going to be scratching the surface so stick around this is going to be a fun [Music] one so why is Docker an important thing to learn in the first place well you see you probably use containers and containerization Technologies every single day of your career if you're a web developer even if you don't realize it containers are a way to build reproducible lightweight environments for processes to run and we use them everywhere in continuous integration and continuous deployment pipelines like on GitHub actions that you probably use all the time and we also use it whenever you're deploying to a server if you deploy something to the cloud chances are you're interacting with container technology somewhere along the way so I think it's a pivotal thing that you need to learn in your web development career but wait what the hell is a container anyways in order to talk about containers we should talk about virtualization now bear with me for a minute because these two things are very closely related and I think it's important to understand the distinctions between the two technologies so let's talk about virtualization so let's draw up how virtualization Works in a typical sense and we can talk through it now typically when it comes to virtualization you start off with a host machine this could be your host now the host could be anything it could be your local PC it could be a server up in the cloud server in a data center somewhere whatever it is it's a piece of Hardware now in this piece of Hardware you have different things that control how this Hardware works you have things like your CPU you also have things like memory and you have your hard drive we'll call this IO now when it comes of virtualization what happens is we take little pieces of each of these pieces of hardware and separate them out into a separate machine this is a virtual machine and then we take these pieces of hardware and in this virtual machine we actually run a full entire operating system now this technique is commonly used in the cloud if you're deploying something to AWS or an ec2 instance typically what you're doing is you're spinning up a new virtual machine that you can then deploy your code onto now virtual machines have a special type of program that can run and manage the life cycle of these machines this program is called a hypervisor and the hypervisor is in charge of virtual machines it manages the life cycle it starts them up it stops them it creates them it deletes them it Provisions resources for for them that is what the hypervisor does now a common hypervisor that you would be aware of is VMware or virtual box these are the programs that control the virtual machines now virtualization is similar but it differs from containerization which is the thing that Docker is kind of based around so let's talk about containerization in a container setup what you would do is you would have a host PC much like the virtualization setup that we set up before now let's say on this host PC we want to run a set of processes but we want these processes to run in isolation we don't want them to touch anything else now we can achieve that using some techniques right let's say we want these processes to run on this machine processes what we do is we can use some techniques like the CH root command which will create a new route for a process so it can only live inside that root and it can't touch anything outside of that like any of the other users directories or things like that that are already on the system we could also use a kernel feature like the r limit feature which will limit the amount of resources these processes take up these techniques amongst other things will en Compass what is containerization now with containerization you could do all this manually yourself but it's really difficult and pretty tricky so there are programs that help manage the life cycle of your containers this is where Docker comes into play Docker is a program that manages the life cycles of containers edit them run them and interact with them so to sum up containerization is the ability to create a lightweight environment where processes can run on a host operating system they share all the same things in that operating system system but they cannot touch anything outside of their little bounded box Okay so we've talked enough about virtualization and containerization let's see containerization through Docker in work let's get our hands dirty so now let's install Docker again this is this portion of our graph right here this is the management layer that will manage the life cycles of all of our containers that we want to create that's what Docker does for us now to install Docker there are examples of how to install it on the docker website for me I use Arch Linux by the way so I going to use Pac-Man to install Docker and it's as simple as that Docker is now installed so how do we know that Docker is even running correctly on our system well Docker gives us a helpful command that we can run to sort of give us our first taste of what Docker will do for us we can do Docker space run space hello hyphen world let's see what happens we're going to break down this command in a little bit but let's just see what happens now okay a lot of stuff just happened let's go through this line by line and let's see what Docker is telling telling us that it did now to start off with Docker was unable to find the image hello world latest now this is the name of our image and this is the tag of our image now in Docker speak an image is basically the thing we run our containers from I'll explain it again in a second and latest is the tag of that image by default doer tags its images with the latest tag so doer was unable to find the image hello world latest locally so it pulls it from a repository Docker will pull images that are already known from dockerhub we can actually check out dockerhub by going to hub. do.com and this is where you can see all of the images that Docker already has pre-built in this platform here Docker hubs this is where you can explore if I wanted to look for like let's say a postgress image here it is right here I can use the postgress image from Docker Hub I don't have to build one myself so Docker Hub is very helpful and that's what it does here it pulls the hello world image from Docker Hub and as you can see here it says status downloaded newer image for hello World latest awesome and then it says hello from Docker this is the actual image running a container I'll explain this in one second here this message shows that your installation appears to be working correctly to generate this message Docker took the following steps the client contacted the docker Damon the docker Damon pulled the hello world image from Docker Hub that's what we were just talking about and then the Damon created a new container for us from the image okay and now the docker Damon streamed that output to the docker client which sent it to your terminal okay let's unpack this a little bit you might be asking yourself what the hell is an image and how does Docker know how to build these things and run containers from images what is all this stuff now before we talk about how to build images and how to then run containers from the images we have to talk about something called a Docker file this is an example doer file and it's pretty contrived but essentially what you would have with any project anything you want to build an image out of is you would have a directory structure that looks something like this in this contrived example we have a directory that can contains a Docker file and within the docker file we have a coffee recipe folder let's just imagine that this is a coffee recipe application of some sort in this coffee recipe folder we have two scripts prepare Beans and Brew Coffee okay so now let's talk about what this Docker file is doing you can see on this very first line we have from Ubuntu latest now you might remember this terminology from not too long ago that means that we want to use the Ubuntu image at the latest tag this is the terminology for an image and the tag for that image Ubuntu latest okay great the next line says we want to run appt get update and appt get install some contrived package what is this this line tells Docker that we want to run something on the image that was added above so we're running apt get update and apt get install on our auntu latest image so this runs command on the image next what we want to do is copy the coffee recipe directory from our local file system into this image so what we want to do here is basically just copy things from our local directory again you can remember our directory structure looks like this we have the coffee recipe directory and under this we have a couple of scripts so we want to copy the coffee recipe directory into this image next we want to run in our image the script prepare beans. sh so this will run that script now remember since we copied the directory into our image we will have this available to us because prepared beans is right here oh look I misspelled it how fun and then next we have this line that says command Now command is the default command that this container is going to run it's this can be overwritten in the CLI but by default we're going to run the Brew Coffee command this is the default command for our Docker image so when the darker container runs it's going to run this default command so now let's zoom out a little bit here what we are going to do is we are going to take in this contrived example this Docker file and from this Docker file we are going to build a Docker image now this image we could name anything but let's just call it coffee right this image by default will create will be called coffee at the latest tag you can give an image whatever tag you want but again by default Docker gives a tag of latest to every image now this image is almost like the file system for the container to run and it's immutable you can only build one image you don't change your images what you do if you want to change anything is you change your Docker file and then build a new version of your image for containers to run now from our image what we want to do is we want to call Docker run and that will then run our container and our container as we said before will call the Run command that we specified in our darker file which was Brew coffee. sh and again to actually build the image we want to run Docker build and there are certain Flags we can pass to build like I said to change the tag of the image from the default of latest you can also name the image whatever you want but this is generally the process for Docker we want to create a Docker file in some repository or in our directory we want to then use this Docker file as the instructions to build a new image for Docker then we can run this image and Docker will spin up a new container on our system that is not able to touch anything else within our system and it will run whatever code we want it to run using the command flag so that in a nutshell is how Docker sets up Docker files images and containers okay so now we have all of our Core Concepts in place we understand the relationship between a Docker file how a Docker file is the instructions to build a Docker image and then how a Docker image is used to then run a Docker container all underneath the umbrella of the docker CLI so let's get our hands dirty and actually try out a real world example this is going to be a simple example it's a contrived example yet again but we're actually going to run a real container on our system to get a feel for how the docker CLI works now let's just say I have a directory called Docker example and in this directory I have two files a Docker file and a print message.

sh file let's take a look at them let's start first with our print message. sh we can see that it is a bash script and then it has a variable that has a list of phrases these phrases are then randomly selected and we print them out to the terminal using a a program called figlet now what if I don't have figlet installed on my local machine well that's okay that's why we have this Docker file so in our Docker file what we want to do as we've seen before we want to use auntu latest that means we're using the latest version of the auntu image from Docker probably from Docker Hub then on that image we want to run apt get update and apt get install figlet and wget we're going to use WG get in the next line where we run W get and we W get some fonts that we want to then install on our system then we want to copy our local print message script into the Container for print m.sh this is what we're actually going to run then we want to chamod plus X print m.sh that just makes this script executable inside of the container and our final command is to just run print message. SSH it's a pretty simple Docker file but this will give us an understanding of how to use these things in real time so now that we're within this directory that has a doer file we can call from within the directory Docker build now in Docker build we could tag that by default it will tag it with the latest release and we can call this asky and we want to make sure we build everything in this current directory this will build our very first image let's see what this command does okay this was a lot of stuff but let's just go through it really quickly and see what we did here we can see that it sent the build context to the docker Damon and then we did the steps that were in the docker file we can see the docker file working for us step one out of six from auntu latest now it pulls from Library auntu this is probably pulling from dockerhub we then run appt get update and app get install these couple of programs onto our abtu image very good we can see that that is running and that's what all this output is and then in our next step step three we want to run W get for these files now these files were then taken and saved into this image very good and now in step four out of six we copy our local print message script into the image then we run chamod plus X which will make it executable and at the very end we want to make sure our default command is just by running this print message script cool okay so now we have our image built how do we run the container based on this image well what we can do is we can check what images we have currently built on our system with the docker images command we hit enter and we can see that amongst a couple others like Ubuntu and hello world we have our asky image tagged at the latest tag with an image ID created about a minute ago and that's what we have right here we built our image I'm going to make my font smaller here because we're printing out asky text and it's going to be pretty large on the screen you'll see what I mean in a second but basically what we want to do is now that we have our image built we can run Docker run and then type the name of the image which is asky we could also optionally add the tag which would be latest and I'll just add it right here if we do this and enter Then This command we can see that it actually runs the container and it runs the script that prints out asky art to our screen isn't that awesome so basically what we've done here is we have now a Docker file that installs STS things into an image we have an image that has all these programs on it like figlet w get and everything else and then it has a script inside of that image as well now the container runs and the container's default command is to then print this stuff out to the terminal and that's what it is we just did that awesome now we can keep running this command we can keep running this Docker container and it will just randomly select another thing apparently it was the same for a bunch of those but it will select random sayings and then print them out in an asky text to us very cool but now let's say we want to modify this script and we want to print out different things what do we do now to update this image well images are immutable what you're going to have to do is edit your Docker file and then create a new image probably with a different tag let's get into that so we see we have our Docker file here and we have our print message. sh let's just say we want to change some of these phrases this is fun I love Docker cool okay so now we've changed the script that actually prints the messages out what we're going to want to do is build a new image from this Docker file because this new image is going to contain different things okay so let's do Docker build- T and what we want to do is we want to name this asky with a colon the colon is going to denote the tag and I'm just going to call it different we can call whatever we want and we want to build it from our local directory now let's hit enter so we can see here we get very similar output to what we had before we probably get less output because we already have the ab image installed and we already have these programs installed in the abutu image that Docker is using to build this image but we can see down below we've successfully built this new image which has a new ID and there's a new tag called asky different very cool all right so now let's check Docker images to see what this looks like okay great we can see that we have as's latest build which isn't currently the latest because we tagged it with something different but that's okay for now and we have our new asky different tag so this tag is a snapshot of our Docker file that was built into an image so now let's run our new image in a container we can call Docker run asky and we want to add the tag of different let's see what happens you can see it pushes out the different text that we did now we have different sayings that we're putting out there like I love you or I love Docker let's make this a little bit smaller so we can see it a little bit better but I love Docker this is fun I love you these are all the different messages that we put in our print message Dosh now the cool thing is we still have the previous version of this image like I said they're immutable you don't change the images you just create new ones so we can also say Docker run asky latest and that was the previous one that we built which has all of these Star Wars things in there let's make this smaller so we can run both of our containers based on the images that we've built the images are mutable you don't delete them you just create new versions of them very cool so what have we learned here well I think we've learned a lot we've learned about the difference between virtualization and containerization and we've also learned about Docker as a whole and how Docker files images and containers all relate to one another but this is just the surface of this surface that we're scratching here if you want to learn more about Docker like Docker compose mounting Docker volumes or even doing things like Port mapping and darker then leave a comment down below of what you would like to learn next and hey thanks nerds [Music]

hey team welcome to this complete course on Docker and containers my goal with this course is for it to be the most efficient way for you to ramp up on your Docker and container skills starting from the basics and progressing through to intermediate and advanced concepts by the end of the course you'll have the Knowledge and Skills to be productive within a team using containers for both development and deployment of their applications you'll understand why you would want to use containers as well as how to integrate them seamlessly into your own workflow this course is split between the and practice for each section I'll cover the concepts that you need to know using helpful visuals and diagrams before diving into our code editor and solidifying that knowledge with real world examples I've provided a companion GitHub repo Linked In the description containing all of the code and configuration that will be shown throughout the course feel free to clone or Fork the repo to follow along as we go before diving in I do want to highlight a couple of prerequisites that will help you get the most out of this course as the image shows it can be tempting to jump straight into learning about containers and skipping some of the foundational Concepts that they build upon I'll do my best to explain everything in such a way that even an absolute beginner would understand but if you have some familiarity with web application development and how all the different components fit together that will be helpful and make it easier to understand what aspects are container specific and what aspects are just General application architecture concerns also having some understanding of basics L shell commands will be useful as we'll use such commands throughout the course if you don't have these I think you can still get value from the course but some things that I do or say might not make sense so it could be worthwhile to go and brush up on those topics first I've broken down the course into the following learning modules each building progressively upon the last I provided timestamps in the description to make it easy to jump between sections if you're already familiar with the topic or if you plan to revisit this video and watch it across multiple sessions in the first portion of the course sections 1 through 4 we'll explore why containers exist in the first place the problems that they solve and the underlying Linux kernel features that make Docker and containers possible we'll also install and configure the docker desktop application and get our first hands-on experience running some containers for the remainder of the course we'll be focused on building out and containerizing our own web application from development all the way through to deploying it to the cloud this is where the rubber meets the road and you'll learn exactly how software companies use Docker throughout the application development life cycle so what is this demo application we're working with I'll go into it in much more detail during section five and throughout the rest of the course but in summary it's a minimal three- tier web application meaning it has a front end in this case built out using react an API layer we actually have two implementation of this one in nodejs and one in goang to highlight how different languages are handled with containers and finally a postgres database each service is designed to be as simple as possible while still showcasing how they all fit together within a real world application when we get to the deployment section of the course we'll learn how to deploy our containerized application to the cloud within the course I'll be using Docker swarm which is a container orchestrator built into Docker and I've also filmed two bonus videos to highlight additional methods of deploying containers one using Railway and another using kubernetes these are separate from the course and information about where you can find them can be found in the description this should give you a taste of the variety of options that we have for deploying containerized applications before we jump into the course content I want to pause for a moment to thank the sponsor of this video Shipyard because of their generous support I can provide this course to you completely free of charge I'm excited about them as a sponsor because Shipyard is highly relevant to everyone working in the container World their platform offers the easiest way to generate on demand ephemeral environments and other words automatically generating fully functional isolated environments for every pull request you connect your G Hub repo add a couple of lines of code to your configuration files and you can have test environments running in just a few minutes later in the course I'll demonstrate how to use the shipyard platform with the course sample application as a bonus for all the students of this course the team at Shipyard has created an exclusive coupon code for you the first 300 people to create a shipyard account and use the promo code devops directive will get 30 extra free days on either the startup or business tier plans links and instructions are in the description below you should go check out the shipyard platform and consider how it can help your teams deliver better software faster without further Ado let's get into the course now in this first section of the course I really want to uh ground our understanding of why containers exist and why a product like Docker has been so successful uh we're going to start with kind of the motivation and then we'll walk through the history of different virtualization Technologies from bare metal to Virtual machines to containers uh and try to understand how that Evolution has occurred and why containers have become the dominant uh kind of development and deployment mechanism for software applications today and this meme on the right is one that I've seen shared a number of times uh it's this idea that I'm building a software application I get it running locally but then something goes wrong in the process of Translating that local system to the production one and so one thing that Docker does really well is help to make it such that our local environment is as close as possible to the production environment that we're going to deploy to and so that's what this is this is hinting at is we're taking that local environment and just shipping it directly to production now there's two main aspects uh of the software development life cycle that Docker and containers are trying to address uh the first one is on the development side so up front as we're building this application maybe historically there was this long process proc you you join a company and they tell you here's your new laptop uh go ahead and and install all these things run these scripts and get everything configured and now you can run the application locally and do your development uh it was very complex lots of opportunity for human error uh and so then Docker came along and Now setting up your development environment is as easy as running a single command Docker compose up we're going to go more into what that is doing later in the course but it took this whole complex sequence of of events that you needed to uh run and turned that into a singular system and command that everyone can run and be up and running uh in the first hour of their their day and so making all of the different combinations of Windows and Mac and Linux development environments all compatible uh is one thing that Docker does really well and containers in general do really well so that's kind of the the de vment side of things on the deployment side of things uh similarly let's say you would build your application and then you need to to put it onto some server somewhere to run it and there was this uh idea that there's a number of steps that need to happen let's say okay we need to create a server running auntu then we're going to run some configuration software like anible to get all the dependencies configured and then finally we're going to copy our application code onto the server and run it with these configuration options again lots of steps lots of complexity uh and much harder to ensure that it's going to be right every single time what containers do is they bring a single standardized package such that as long as my server has a container runtime and there's a number of those available Docker provides one but there's a number out there all you need to do is take this standard container image and run it with these specific options and that's it so now instead of worrying about the underl operating system and the underlying dependencies as long as we have our system and a container runtime and our container image we can run it and get repeatably reliable results uh over and over and so these two aspects are really why containers exist and you can see how much simpler it gets when we use containers going from this historical system with lots of steps and and potential for error into a much more simplified uh unified approach using a system like container and like docker and so that hints at what a container is I've talked about kind of the standardized package uh but from the docker website a Docker container image is a lightweight Standalone executable package of software that includes everything needed to run an application so you can think of it as this box that contains the underlying sort of os dependencies let's say runtime dependency so if I was writing an app in Python it would have the python runtime with the version that is needed uh it would have any of the Python libraries that I'm using so maybe I'm using SQL Alchemy to talk to a database or fast API to build out an API and then it's also going to have my application code so the specific application code that I'm writing all of that gets bundled into this container image and then the beauty is that we have this standard interface and can run multiple copies of it so let's say I need two copies of this application I can take that image with a container runtime and run a container run two copies run three copies Etc and so I've used the term container and container image it's important to know the difference there a container image is this artifact that has all of the all of these dependencies within it a container is what you run from that image so you take an image and if you think of object oriented programming a container image is like the class and then the container itself that you're actually running is like an instance of that class an instantiation of that class and so a container allows us to have this standardized Packaging that we can then create one or more copies of uh and those copies are going to be the same every single time so far I've been using the term Docker container uh but I want at this point to call out what is known as the oci or the open container initiative and effectively there were a few early players including Docker building out the early implementations of containers and as they became more popular companies realized that rather than having a whole bunch of slightly different but similar things that we're calling containers it would make much more sense to come together and create one standard that can be used across different implementations that's exactly what the cont the open container initiative is a number of companies including Docker Google VMware Microsoft Dell IBM Oracle the list goes on came together uh and created open industry standards around what a container is and how it should be run and so there's three primary specifications that the oci defines the runtime specification the image specification and the distribution specification so the image specification defines what should be included in the image in terms of metadata and the format that it should contain so a serializable file system the runtime specification defines how you would take an image that adheres to that image specification and run it in a container and then the distribution specification talks about how those images should be distributed so think about Registries and pushing and pulling images and so these three things together make up the open container initiative and Docker has a specific implementation of this standard so when I talk about Docker images or Docker container images what I'm referring to specifically is the docker implementation of this oci specification if Docker is the only container tool that you're using it's not all that important to know this I wanted to provide this context in case you start using other tools and know that there is this standard format that can be shared across different Tooling in the ecosystem now now let's walk back in time a little while and understand how we got to this idea of containers hopefully from my description of the motivation uh you understand why containers are useful but it's I think it's also important to understand the history and the context of how we got to where we are today and so before there were containers before there were even virtual machines uh we just had computers uh we now call this bare metal Computing so if you're running your applications directly on the Hardware system itself uh another term for that is bare metal but back long enough ago this was just called Computing right so in this case you have your your physical host your server um the physical Hardware layer at the bottom you've installed whatever operating system you want on top of it then you've installed whatever binaries and libraries are necessary to run your application and then you would have your application one or more applications installed directly uh onto this host machine now what are some challenges with this setup uh first of all you can see that shared layer of binaries and libraries uh it's shared across both of those applications and so if your applications for example need different versions of the same dependency or have some dependencies that are somehow incompatible with each other now we've got we've gotten ourselves into what is known as dependency hell so uh we somehow need to either modify our application such that those dependencies are not compatible or figure out a way to to carve up the system such that we can have both versions of the the libraries available doing things like setting uh your your path environment variables differently for the different processes Etc so that can be quite a challenge um you also because of this dependency management challenge you're more likely to have le fewer applications on each host machine uh and so it makes it harder and harder to achieve High utilization so let's say application one takes one CPU core application 2 also takes one CPU core but we have this server-based uh system that has 64 cores um it could be very hard to sort of utilize that compute capacity effectively because of the challenges uh associated with managing dependencies for for applications you also now if you are installing multiple applications onto the system if you ever have an issue with one of those applications or do something that will mess up uh some underlying dependency now you have this large blast radius meaning uh changes or or issues with one application can directly impact uh the performance of another application running on that same host machine so uh it just makes it much more dangerous to to modify things also the startup and shutdown speed of a physical system is on the order of minutes so uh to spin down that server and reboot it for for some reason uh that can take a number of minutes and so so it's just relatively slow if you want to try to achieve high uptime and and eliminate downtime for your applications in addition to that creating new physical machines uh sort of provisioning a new system is very very slow so depending on my setup if I have my own Data Center and I need to actually order the hardware and get it installed and hookup networking that could be on the order of days or weeks if I'm working with a co- location facility or a provider that offers bare metal systems that provisioning might be on the order of hours um but still quite slow relative to what we'll see as we evolve and and sort of add in this concept of virtualization and so that's bare metal I think most people are familiar with working with bare metal um even if they don't realize it because when you run things on your laptop or your desktop you're installing them directly on the host and so that is this concept of of bare metal however that's not the only way to do things and so these challenges really prompted Engineers to come up with this concept of virtualization and adding virtual machines such that we can eliminate some of those challenges and so in the case of a virtual machine now this outer boundary is still the host the physical system the server uh we've got our Hardware um we may or may not have an operating system running on that Hardware I'll talk about that more in a second but then we have what's known as a hypervisor and so a hypervisor is some combination of software and Hardware that allows us to carve up the physical resources so think CPU cores memory access to storage or gpus other devices it allows us to carve up that pool of resources into isolated smaller pools that we can then install our our systems onto and so this hypervisor kind of gives us this virtual Hardware layer upon which we can then install an operating system install binaries and libraries and install our applications and so now rather than having both applications on the single physical host we now have application one and application 2 each with their own virtual isolated virtual machine uh running on top of that hypervisor and so there's two types of hypervisor type one means there's no underlying OS these are running directly on the physical Hardware um this is beneficial because you don't sacrifice any performance in that sort of os to software virtualization layer uh there there's a small performance hit but much less than if you have an operating system on the hardware and then a hypervisor on top of that uh so things like AWS Nitro so that's the hypervisor that Amazon web services is using to create virtual machines on their servers uh VMware is a very popular virtualization uh company and they have vsphere which is a type 1 hypervisor if you want to just create a virtual machine let's say on your laptop you're probably going to use a type two uh hypervisor uh one such example is virtual box so you can install this program virtual box and then within that you can create virtual machines for various uh operating systems Etc and so how does this compare to those challenges that we saw with the bare metal systems uh one we've eliminated this dependency hell so now each virtual machine has an isolated set of dependencies binaries and libraries such that our application can specify whatever dependencies it needs and not have to worry about the needs of other applications uh because of this we're going to have better utilization so we can carve up a big server CPU into many different smaller uh chunks and each of those chunks can have its own virtual machine and then we can install our applications accordingly so this is what enables a cloud provider you can go on and see oh I can I can choose between all these different machine types they don't actually have all those different machine types installed in their data center they have much larger machine types and you're getting some sliver of that installed as a virtual machine you also we've also reduced the blast radius significantly such that now if if something goes wrong with application one on virtual machine 1 that should be isolated from application 2 and Virtual Machine 2 uh and it should not cause any issues and so because of that we've we've increased the safety of of making changes we've also sped up the speed at which we can start and stop these things so now it's still on the order of minutes probably um but faster than shutting down and starting a physical server and then finally uh uh one area where virtual machines have a big uh bonus over bare metal is that we can provision and decommission them much much faster and so we're all used to living In This Cloud world where we can go onto our favorite cloud provider click the button or issue a command and have a new virtual server ready uh within minutes or if we're done with it we can decommission it in a can also be done and and we can stop paying for it within minutes as well and so that's the sort of the second tier or the second rung along this path of virtualization the next one being containers and so containers obviously are the focus of this course um they change things a bit further still so now in this outer box we've got our host or our which could be a physical system uh it also could be a virtual machine so we can take either of those two uh Concepts from before and we can run containers within that context uh you can see at the bottom we've got either our virtual or physical Hardware with an operating system and then the key piece here that's kind of analogous to that hypervisor uh is the container runtime so this is a piece of software that knows how to take a container image and run it within the host system uh the key difference between virtual machines and uh containers is that virtual machines are running their own copy of the Linux kernel uh and I'm talking only about Linux containers within this course there are windows containers but that is a different topic for a different day and so with Linux containers uh they are actually running they're sharing the kernel with the host operating system uh versus in a virtual machine you have your own oper you have your own kernel and you have a copy of your own operating system that's running and So within a container you're sharing that with the host however we're using certain features that'll go into later of the Linux kernel to help provide uh isolation between the different container processes and so you don't get quite the same level of isolation that you would with a virtual machine you do get quite good isolation if you configure things properly as we can see now we have our two containers both running uh on the same host however they are still isolated and have their own copies of binaries and libraries even though they're sharing that underlying kernel now there's a couple of different sort of types of software related to containers the first category of which I'll call the desktop container platforms so these are sort of container Management Systems things like Docker fall into this category things like podman these are things that you would install on your development system to manage containers manage container images uh run containers uh set up the the necessary networking Etc um so these are kind of the holistic lifecycle systems and then there's much more targeted or specific pieces of software called container run times which include things like container D or cryo and these are very specifically designed to take a container image that aderes to the spec and run it uh with the necessary configuration options to provide that type of isolated environment that that is shown in the image now how does this compare to Virtual machines and bare metal in terms of those challenges we saw before again we don't have the issue of dependency conflicts because they each have their own underlying binaries and libraries uh We've improved our utilization efficiency even further because we now for each virtual machine we had to have an entire copy of that operating system so each of those virtual machine images is going to be on the order of gigabytes whereas container images can be much much smaller because they're able to leverage that underlying OS and so they're generally going to be on the order of megabytes uh I have seen very large multi-gigabyte container images but that's not ideal uh but because of this we're able to utilize the resources of the system much more effectively again we have a small blast radius where we have some level of isolation even if it's not quite as strong of an isolation as a virtual machine we're able to start and stop containers even more quickly than virtual machines so on the order of seconds uh we're able to in this case starting and stopping a container uh is kind of equivalent to provisioning and decommissioning so also on the order of seconds we can bring up a new container from scratch and clear it out as well and then the final sort of key piece I think with containers is that they're small and light enough that we now start to use them within our development environment sure you could use something like virtual box you could create a virtual machine that tries to replicate your uh your production environment but it's kind of slow and clunky and not that many people do some people do but but many people don't containers are light enough uh that now more and more will start to actually run containers and we're going to do this later in the course run containers within our development environment such that we can have high confidence that our our production environment is as similar as possible to that development environment and that just decreases the likelihood that we're going to hit some small nuanced bug uh between the differences there uh and so we're able to develop more efficiently uh using a technology like this and so I I pointed out here that this host can be a virtual or a physical machine and so now we can get a little bit uh Inception with it where we have a host machine with virtual machines running on it and then containers running within uh those virtual machines and so this is a very common configuration that you'll see uh most people who are working with containers in the cloud are going to have something like this where you have multiple virtual machines each running multiple containers uh and so this is just combining all these Concepts together and the real challenge now becomes how do I how do I manage all of these containers across those virtual machine hosts and that's where this concept of orchestrators comes in and so things like kubernetes Nomad from hashicorp or Docker swarm are these systems that are designed specifically to take containers and run and manage and schedule them across multiple nodes and so in this case the node is each of those virtual machines and so when we get to the deployment section of this course we'll talk a little bit more about this but it's just important to know kind of the concepts of hey I'm starting with this bare metal server it my my code has to run somewhere on some physical CPU I'm carving it up into these smaller virtual machines that I'm able to manage a little bit more effectively but then I also want to gain the benefits and the speed uh and the efficiency that a container provides and so combining these different types of virtualization in this way that we see more and more companies using in this day and age this just summarizes most of those challenges and tradeoffs uh that I talked about before uh in terms of dependency management uh virtual machines and containers really shine uh in terms of utilization containers definitely win out uh just because it's much easier to get uh to to pack applications more efficiently uh when running them in containers uh from an isolation perspective uh you can't get any more isolated than than multiple physical hosts if you're running application a on on one server and application B on one other physical server that's the strongest isolation you can get virtual machines still provide very strong isolation and those hypervisors are these very optimized systems with a small attack surface area that have really been battle tested and so I I think isolation provided by virtual machines is almost as good as as those from bare metal obviously there could still be some unknown bug that that causes uh a security issue there but I think most people have quite high confidence in the isolation that hypervisors provides and then containers um this is one area where they are less good than these other options is they do provide some level of isolation we'll talk about the the ways in which they do that but they don't give the same they're not as strong and if you make a configuration mistake the the level of isolation can be dropped significantly so that's why I gave containers a yellow on isolation um startup speed containers win out the ability to run containers within a development environment because they're lightweight enough means that we're much more likely to do so and therefore have stronger parity or have have our development environments and our production environments more similar to one another this control row is one that I haven't talked about as much this is just the fact that you can have full configuration control over the system um in a bare metal context that you might not get in a virtual machine so only you only have control over the features or the specific configuration options that the hypervisor passes through or the the container runtime passes through and so maybe there's some specific type of GPU or Hardware accelerator that is not yet supported by your particular container runtime or your hypervisor in those cases the bare metal system is going to win out because if you need that specific Hardware config that might be the only way in which you can achieve it and that leads into kind of this this next row of performance obviously ly a summary of red yellow green um here can't fully capture the nuances you don't take that much of a performance hit when you move to a virtual machine or container however I think one place that performance can win out on bare metal is because of that control you're able to do things like if you're if you're running multiple servers for example ensure that they are directly next to each other on the same server rack such the networking latency is as small as possible or ensuring that the storage devices that you're using are directly connected to that physical Machine versus accessing them over the network and so there is Performance Tuning that bare metal provides the opportunity to do um but overall the performance is fairly comparable for most use cases across these different Technologies and then the final column there just sort of how challenging it is to work with and so so if I'm if I'm buying and racking my own servers in a facility that's intensely difficult to do and and manage there are now providers that can give me access to Bare Metal systems and handle some of this operational overhead but I think it's still more challenging than virtual machines and more challenging still than containers where I can spin something up and and down very easily and there's all of these services that have container run times such that as long as I provide a compatible container image I'm able to run it within that system and we'll see that optionality uh in in a later portion of the course when we get into deploying our own containerized applications and so now that we have a better understanding of the motivation and history of containers and virtualization Technologies I think it's useful to do a bit of a technology Deep dive and start to understand the the foundational Linux kernel features that software like Docker is using to make the magic of containers happen and provide those isolated environments that we described uh then towards the end of this section we'll actually take a look at the docker system architecture itself there's three technologies that make up the core of uh a container and that is namespaces control groups and Union file system and now namespaces are a a mechanism to wrap a global system resource such that from within that namespace any processes running within the namespace it appears as though they have their own isolated instance of that resource uh and they cannot see things happening outside of their own instance uh so for example on the right here we have a listing of the various types of name spaces at one very good example example is the PID namespace so the process ID namespace within a container it is isolated with with a p namespace such that the process I'm running appears to be process ID number one even though there may be tons of other processes running on the host from inside the container we don't know anything about those host processes and we essentially think that we are the only thing running uh in this system another good one to call out is the username space uh so by using a username space containers are able to map a one user within the container to a different user on the host system so we could be running as a root user with elevated privileges inside the container but by run as long as that container is run within a user name space that's configured correctly that root user inside the container may actually map to a non-root user outside the container uh and that's very important from a security perspective in case there was an issue with with one of our containers and an attacker was able to gain access to it and break out of that container if we had properly set up username spaces then on the host they would not have the same level of privileges that they had inside the container cgroups are a Linux kernel feature that allow you to organize your processes into groups which you can then limit and monitor access to certain resources and so on the bottom left you can see within the file system under proc cgroups we can see all the different cgroups that are available these correspond to different system resources whether they're CPU memory storage devices Etc and using cgroups we can specify a configuration for a particular process or group of processes on how they should be able to access those system resources and so for example we might have one application that is enabled to use up to 30% of the CPU Cycles we would use the cpu. shares cgroup maybe it can use up to 50 megabytes of memory we would specify that with the memory. liit in bytes cgroup and then we could also limit the amount of disk read that it has access to uh using the block IO throttle read bits per second device C group uh now the reason that this is important is that it allows us to avoid what's called a Noisy Neighbor problem where one application uh is very resource hungry and sort of starves the other applications of resources by using cgroups we can isolate from a performance perspective each of our different processes and each of our different applications now the third key underlying technology is what's called a union Mount file system a specific example of which that Docker uses is called overlay FS it essentially allows you to take separate file systems and combine them into a unified view that you can then operate on and so in the image you can see we have a lower file system and an upper file system with various files we got file one 2 a and four in the lower layer in the upper layer we have a different file 2B a new file three and then we've actually deleted file 4 when we combine these layers into the overlay we end up with a a combination of those two things where the upper layer takes precedence over the lower and so we have file one file 2B and file three and so while Nam spaces and cgroups are sort of what make containers possible it's this concept of a union Mount file system that makes containers practical because of this layered approach we're able to share lower layers and cache those layers such that the amount of data that needs to be transferred and stored for container images is much much smaller because many images can share a common lower layer and then only some subset of the upper layers can be modified which enables us to move less data when we need to get a container image onto a system and allocate less space if we're running multiple copies of the same container and so with those Technologies while you can interact with them directly it's software like Docker that takes it and packages it into a really nice user experience so rather than needing to know all the different commands to create control groups and move processes into them or create Nam spaces and and block off the different portions of our system in such a way Docker has bundled that uh into an easy to use desktop application that we can install in our system and get all of that uh with a series of simple commands that we're going to learn throughout this course and so this is a diagram of the docker desktop application uh there's there's a number of pieces here here so let's call them out in turn on the left hand side is the client application and so this is the thing that we will be interacting with there's a a number of elements there's the command line interface so when I go to my command line and type Docker run or Docker pull I'm interfacing with that Docker CLI there's also a a graphical user interface which gets installed uh this is nice for browsing which images you might have on your system uh you can do things like configure how much CPU and and memory and disc space the docker application has access to so there's a number of nice user experience features that the that the guy provides there's also a credential Helper and so this is when we learn more about Registries later on in the course essentially a registry is a place where we can store we can push images to and pull images from and so in order to have a private registry you need some way to log in and so the docker credential helper is a mechanism to help store the necessary credentials to do the authentication for that and then finally uh extensions are a relatively new feature of Docker that are thirdparty pieces of software that plug into the client and provide additional functionality in some way and so that's the client application when you install Docker desktop on your system it also creates a Linux virtual machine uh so if I'm on Mac OS there's no such thing as a Mac OS uh container uh instead Docker uses a hypervisor installs a Linux virtual machine and then within that virtual machine it setss up uh two things it sets up the server host application and so that's known as the docker demon or Docker D that exposes an application programming interface the docker API and so when I execute a command from the the command line uh let's say I do Docker run that command gets passed to the docker demon which is listening via that Docker API and then it executes whatever command that is specifying uh within that server host application so let's say I did Docker run and then specified I wanted to run my container image one I might create it would then go off and create that container 1A let's say I then issued that same command Docker run with my container image number one again I get another container container 1B and so this Docker demon is managing those container objects as well as things like uh networking and and volumes all within this server host application finally there's an optional kubernetes cluster that gets configured and installed uh within that virtual machine this can be a nice to have if you are developing four kubernetes you can leverage that cluster directly without having to uh install and configure a separate cluster I mentioned for Mac OS it's a Linux virtual machine on Windows there's actually two options you can either configure it to use uh WSL Windows system subsystem for Linux as the back end or you can use hyperv as that virtualization layer uh to create a virtual machine I think either of those are viable when we go through the installation and setup in the next portion uh we can talk through that in a a little more detail and then the final portion to call out on the right I hinted at this but a registry is just a place to store your container images and share those container images uh with your team or with the world or with wherever you're deploying uh your containers to and so while it is not a part of Docker desktop it's useful to call out here uh because that Docker demon when it needs to get an image that it's not building locally it's going to go off to some registry dockerhub is a great example of one such registry and pull that image into the the host environment so that it can then run the container from it there's also a little bit of confusion around Docker Docker desktop and Docker engine I've highlighted a few of the subcomponents here that make up what's known as Docker engine and so that is the command line interface so that Docker CLI uh as well as the docker demon and the docker API those make up what is known as Docker engine and that is the open-source core of Docker because the the Docker demon only runs on Linux you can only install Docker engine on Linux you don't get sort of this niceness of the Linux virtual machine and sort of the developer experience that the GUI provides but if all you need to do is build run and interact with containers then Docker engine might be sufficient for your needs in this course though because we're using Docker for development and likely many of you are on Windows and Mac OS systems we're going to install Docker desktop so that we can leverage all of the developer experience niceness that Docker desktop provides at this point I think we are ready to go ahead and install Docker on our systems and start using it one thing that is important to call out is that for personal use Docker is completely free you can use Docker desktop without paying anything uh however if you're using it for commercial purposes and your team has either more than 250 employees or more than $10 million in annual revenue uh it does require one of the paid tier subscriptions so depending on how using it within your company uh it may or may not be free so that's just something to call out before we uh before we install it but for personal and learning purposes it is completely free now to install it we can just go to the docker docs under docs. do.com Docker and so in preparation for this video I actually uninstalled Docker so I'm going to go ahead and pull open Chrome and go to that uh site and so here I am at docs.

do.com getd doer it gives us options for Windows Mac and Linux uh so choose whichever operating system you're on and then it will have some additional system requirements uh and information that you're going to need I'll just click Docker desktop for Mac with Intel and it'll start downloading uh it is quite a large file so we'll see hopefully my uh home internet is fast enough down below it has information about system requirements uh for depending on the type of system that that you're running so you just want to go through uh and read these the installation process is pretty much just like any other desktop application here it's downloading that dot DMG file I'll then open it drag it into my applications folder on Windows similarly a pretty straightforward installation process the one thing to call out is the two different options for the back end so uh I described in the architecture diagram how you have your client running directly on your system but then you're going to have this virtualized environment where the docker demon is running and so this is choosing whether you want that Docker Dage to be running within WSL or using uh hyperv as the the backend system if you are on Windows I would probably just go with this WSL backend as that's kind of the default option here so it looks like my installation file has finished I'll drag it into applications it expands from the 600 megab download into a 2 GB install and now I should be able to just go to my applications and click on Docker it's verifying it asks whether I want to trust it or not click open and so now here is that graphical user interface uh one thing that I will call out is you may want to go under settings and then resources and here you can specify how many of your system resources that Docker as a whole should have access to so here it's defaulted to four CPUs 10 gigs of memory I'm going to give it access to a little more memory uh the defaults are likely fine for just getting started but if you find yourself running up against resource constraints modifying these uh resource configurations can be very helpful in in alleviating those constraints in order to have those changes take effect you have to click apply and restart and so I'll do that now now at this point we can run Docker commands and installing that application gave us both the CLI as well as the the docker demon and so let's run our first container um one fun example that we can use is uh the whalsay container and so I can do Docker run Docker SL whales say uh and then the command is going to be C say and I'll just say hey team and so if I run that it's going to go off and find that public whale site image download it from dockerhub so we can see it downloading now and each of those lines corresponds to one of the layers in that overlay file system and so it's downloading each of those images onto the host system such that we can then run a container from that image and we see it took the phrase that I put in here uh and gave us this nice asky art output uh with that phrase built in and so with that we have uh pulled an image from dockerhub run a container from it and provided a custom command so that's kind of the the most basic thing that we can do uh with a Docker image and so that's a fun example but let's do something a little more useful let's say we needed a copy of postgres 15.1 running uh for our application uh we can just issue a Docker run command and then I'm I'm doing two options here one I'm setting the postgress password environment variable uh that's just something that's needed the container won't start if you don't set a password and then I'm also publishing Port 5432 and so because these Docker containers are running on an isolated Network the only way that I'm going to be able to connect to it from my host is to publish that port and so that publish command is saying take Port 5432 on my my my Local Host my system and connect that to Port 5432 inside the container uh I'm then specifying the postgres image and specifically uh version 15.1 uh running on the Alpine operating system again it didn't have that image locally so it's going to pull that from dockerhub once it finishes pulling it then starts up the image and now we have the database system running inside that container ready to accept connections uh just to show that it is actually running and and accept connections we can open up PG admin uh and we can see I've added this Docker database uh and it was able to successfully connect to it uh and we can for for for example run a query I could do select star from information schema tables uh and run that and we get all the tables from that information schema within the database running inside the container hopefully this gets you excited about the possibilities of what running inside of these isolated container environments can provide in terms of your development life cycle now that we've run a couple of simple containers on our system I want to really dive deep on using thirdparty container images and how we can leverage these within our development process to solve a number of different use cases now dockerhub is a container registry hosted by the company Docker there's over 100,000 publicly available images ranging from things like Alpine and auntu which are different flavors of Linux to specific tools such as engine X which is a a web server and reverse proxy or busy box which is a utility image that has tons of different uh useful commands that you can leverage on your system from within Docker if you go to hub. do.com you can then search the images or browse based on the different filters there on the left and so if there is some utility that you want to use likely someone has created an image for that now before we go off and create a bunch of containers on our system I do want to pause and call out the fact that by default any data that we create or modify within a container at runtime is going to be ephemeral meaning if that container is stopped and removed from our host all those data will be gone now there are two mechanisms by which we can help to solve this uh for dependencies that we want to be in the container when it's started up we'll want to build those into the image itself and we'll look at how to do that um as you can see in the diagram on the right if we remember that overlay that Union file system approach we have all of our different build layers there and those at runtime are treated as readon so we've we're starting with an auntu image we're installing uh some dependencies we're copying in our source code we're running npm install to pull in those node uh those node dependencies all of those layers when we create a container from that image or treat it as readon and then there's just this thin read WR layer on top that's where any data that's modified at runtime will go if there's some additional dependency or configuration change that is needed rather than starting the container and then making that change where that change would live in the the the read WR layer at the top we'll want to pull that into our build process such that every time we create a container from that image it will have the necessary change now the other case with data that we may want to persist are data generated by the application at runtime so this could be things like data that we're storing in a database uh when a user creates or or modifies some uh some aspect of their profile let's say on a website those data we can't incorporate into the image because they're not known at build time but instead we need to store them in such a way that they'll be that they will persist beyond the life cycle of the container itself now there's two mechanisms to do that the first of which is called a volume and so we can create what's known as a Docker volume that will designate a particular uh location within uh the docker host system so it will be within that virtual machine that Docker is running uh located at/ barli doer volumes uh we can then when we run a container tell Docker to use that volume and mount it at a specific path within our container file system such that any data we store there gets stored in the volume and that can be managed separately from the life cycle of the container itself now the second option is a bind Mount it's a where the volume Mount the data is still stored within that virtual machine on the Linux file system with a bind Mount we're actually we're actually connecting our host file system into the Container at runtime so I'm going to have some path on my system uh and then I'm going to mount that into the Container at runtime and now if I modify any files within that path they'll show up on the host file system both of these are viable options Docker suggest using volume mounts by default one reason for this is that because that bind Mount is crossing the boundary between our virtual machine and our host system performance can be much lower if we are doing things that have a lot of reads and wrs and so by default use a volume Mount if you need to have easy observability into the data that's being modified or you want to be able to see the files that are being created more easily then a bind Mount can be a good option also later in the course we'll use a bind Mount to actually Mount our source code into the Container such that when we make changes and are developing on the Fly those changes can be represented within the container file system right away I just covered a lot within this one slide but I created a walkth through in the GitHub repo that we'll do now that I think will really solidify these Concepts so the read me for the fourth module of the course uh so the running third party containers module has all of this within it and so now I'm going to walk through the process here that is outlined in this read me and I think it will make what I was just describing uh much more clear about data persistence so remember our container file system looks like this we have a number of readon layers that that exist in the image and then we have a read right layer at the very top let's do some experimentation with uh running a container installing something in it and seeing how that behaves when we create additional containers from that same image so let's start by doing a Docker run command and we'll pass it the uh-- interactive flag in the D- TTY flag those two flags together mean that when we issue this command we'll get a running shell within that container the-- RM flag tells Docker that once this container process exits so once we exit the container uh it should not store that stopped container it should actually remove that from our system so if we run this it's going to go off and pull that Ubuntu 224 image then it starts the container and now we're running a shell inside that container let's try to Ping google.com from within the container as you can see the pin command is not found it's not installed by default within that UB 2 image to install it we can use the APT package manager we can do apt update that'll go off and fetch all the metadata associated with uh the APT package manager and then we can do AP install IP utils ping I'll say yes and so now with ping installed we should be able to call Ping google.com uh one time that's the- C1 it's successfully pinged Google uh and that's great now I'm going to exit this container and if we want to take a look at the container ID here d61 we'll get a different container ID the next time we create create one so now I'm going to rerun that same command we're going to create a new container from that same image and you see we have a different container ID this container doesn't know anything about that previous one and so because we installed that pin command within the readr container image of the other container it was blown away completely when we exited and Docker removed that container so now that we're in this container again if we try to do ping ping is not found however let's change this experim ment just a little bit so I'll exit this container and now I'm going to run the same command but I'm going to add I'm going to remove this dsrm flag so Docker is not going to throw away the container after it stops and then I'm going to add a name option and so I'll name it my Ubuntu container and now we have a shell inside that container again we're going to do apt update and apt install and that will get our ping utility installed within the the the container okay so now with ping installed we should be able to use it great and so now when I exit because I didn't have that D- remove flag the container still exists on my system if I do a Docker PS that will list containers but because it's stopped it doesn't show by default so I'll do a dash a and we can see my Ubuntu container was running it was created 41 seconds ago it exited 13 seconds ago but it should this container even though it's in its stop State still has that read write layer associated with it uh so I can uh now restart my container with the docker start command so I'll do Docker start and pass it my container name and then I can use the docker attach command to attach to the Shell running inside that container so now I'm back inside the container and I can ping Google and it works successfully so that just shows we created a container we installed the dependency in that container layer we stopped the container but did not delete it and then we were able to jump back in that same container and that dependency existed again now normally we would never want to jump into a container and modify things like this on the Fly the better approach and the approach that we'll want to use for a dependency like this is we want this to always be there when we start a container from that particular image so instead of starting the container and resisting it uh in the container itself instead we want to build our own image containing that dependency and so we're going to do a much deeper dive on building containers and optimizing those builds in module 6 but we can write a very simple Docker file here and pass it at the command line so this is just saying start from that Ubuntu container image run the APT update apt install and that passes it to the docker build command and then we're going to tag it with this tag my Ubuntu image so if I run this command Docker is going to build a a brand new image based on the original but now this new image is going to have IP utils installed out of the box if I want to run a container based on the new image I'll use the tag specified in the build command and now I don't have to install ping because it's already installed so now I can run ping Google and it succeeds so hopefully that gives you a better idea of how this container file system works and the fact that if we want some data likely in the form of a dependency or configuration to be available at runtime every time we want to build that into the image we don't want to start an image and then make modifications to get it to the state we want and so that's the that's the case of data that we know about at build time and can build into the image directly the Second Use case that I described was Data produced by the application at runtime uh and so like I mentioned this could be data that's going into a database let's do some experimentation to build up some intuition about this type of data again I'm going to create a container from that auntu image now this time I've shortened the-- interactive D- TTY uh you get the same thing in a shorthand version with just - it and so here we have a shell running in that container to simulate uh data created at runtime I'm going to make a directory called my data and then I'm going to Echo hello from the container into a file within that directory and so now if I C my data hello text we can see hello from the container stored in a file on the container file system if I exit the container run a new container from the same image what do we expect if I try to do cat my data hello text of course that file doesn't exist just like when we installed something into the file system it's in that top readr layer that gets Blown Away away with the container but now we can no longer build this into the image because we don't know what this file should say ahead of time so instead we're going to use a volume I can run Docker volume create my volume and that creates a volume and now I can modify my run command such that I mount the volume into the file system of that container so I've added this-- Mount flag it's using the source of my volume which I just created and it's using a destination of my data root SL myy dat now I'm going to do exactly what I did before I'll do make my data oh it already exists because I mounted that volume that directory was already created so I'll Echo just like before hello from container into a file within that directory I'll C it our data exists within that file system but if I exit and then recreate a new container also mounting that same volume we can go into my data and see that hello Tex has persisted across that container destruction uh so the data was stored in the volume even though we stopped the container and Docker removed it from the system the volume is treated separately and so we were able to persist the data across that container boundary now you might be wondering where where does this file actually live um it has to live somewhere on disk and in this case because I'm on Mac OS it lives within that virtual machine that Docker is managing um so it's if I was on Linux it would just be directly at VAR lib Docker volumes because I'm in Mac OS I don't have VAR Li Docker volumes that lives within the virtual machine one way that we can take a look at that is to run this privileged container so this is a Nifty trick that I saw online where I can run this command and it will essentially give me a shell into that virtual machine that Docker is running and now I can navigate to VAR lib Docker volumes I see a whole bunch of volumes the one I care about is the one that I named my volume now if I go into the data directory if I go into the data directory and I see my hello. text and so this is showing in that virtual machine file system where that volume actually lives now this was just a trivial example where I was creating that file and wanted it to persist let's say we were running a database like we were running postgres before we would want to use the same approach with a volume and mount it into the file system of the container where postgres was going to store its data and so by default postgres stores atar lib postgres data and so we would want to modify our Command to use that mounting syntax uh there's an alternative syntax with the- v that does the same thing and this would allow us to store the databased data in a volume and manage that separately from the container now I mentioned that volume mounts were one option and bind mounts were the other let's explore what a bind mount looks like for the same use case now I'm going to take my command that I used before where I had a volume Mount and change the type to bind so now the type is bind my source is my present working directory and in particular the my data directory within my present working directory and so let's just list out to show I have a my data directory here within there there's already hello. text let's remove that so now there's nothing in my data and if I run this command I'm going to mount that directory from my host file system at the present work directory SL myy data and the destination is going to be rooty data on the container so let's Echo something in here blah blah that's going to go into hello.

text so inside the container we have blah blah within hello. text if I exit that container I'm now on my host system and if I cat my data hello. text we can see that file was created and stored at the path that I specified I can also modify the host data and that will be reflected inside of a container mounted in the same fashion so I can do I'll add another blah okay so it's three BLS outside the container I'll now recreate my container it'll actually be a new container but still mounted at that same path and if I do cat my data hello. text we see the updated version reflected inside that container and so as you can see bind mounts just like volumes are a mechanism to persist data beyond the container life cycle uh the volume Mount is kind of tucked away inside the the file system that Docker is managing the bind Mount is located on my local system by default Docker suggest using volumes they're slightly easier to manage however if you want easier visibility into the data itself it can be convenient to use a bind Mount just be aware that you can suffer some performance impacts if you're doing heavy read wres into the bind Mount location so my hope is that this has helped to build your intuition around how data is handled within containers uh both sort of the default case then the case where we want dependencies available in the container when it's started and then the third case where data is generated at runtime and sort of how we should manage each of those either by building those data into the image or by mounting a volume or a bind Mount uh to capture those data beyond the life cycle of the container the other thing that I've called out within this read me that I think you'll find useful are specific use cases for leveraging thirdparty containers directly and what by that I mean not needing to build our own container image the first one is databases so databases are always a pain to deal with sometimes you need one version sometimes you need another sometimes you'll have multiple databases that you'll want to interact with for different projects Docker provides a super useful mechanism for spinning up a new container with a different database that you can use and have that completely separate from one application to another from one version to another uh very very easily I think the key considerations are to always use a volume to persist the data um if you have additional configuration that you need so let's say there's a config file that you need to have in the container when it's starting up to to in order to set some specific configuration you can use a bind mount to mount that from your host file system into the container and then finally you'll want to set the necessary environment variables so a number of the the official database images use environment variables to configure things like the root password within the container so earlier when we started that postgres container we had to set the the postgis password environment variable uh there's a number of things like that with these database container images I've laid out a number of a number of different databases and I've tried to capture the specific volume mount and environment variables that you might need to use so we've seen postres before um here I've added that volume amount for the location where the data gets stored uh if you had a custom postr SQL comp file and didn't want to use the default you would modify this command to use this bind Mount so you would have this on your local file system and then you would pass the- C flag uh to The Container when you when you run it and pass it a path to that flag flag so this is as you can see where we mounted inside the container this is that same path and that tells the container upon startup to look there for the configuration file so mongodb the official image it looks quite similar as you can see it's expecting to save the data to SL dat DB so we're we're putting that into a volume uh we're setting the initial root username and password with environment variables and we're publishing the default Port of 277 again if we have a specific configuration file we need to pass in we're going to bind Mount that into the container and then we're going to pass a configuration flag to tell where to look for that I'll let you look at the rest of these on your own time but I've I've laid out the commands for redus MySQL elastic search and neo4j now the Second Use case that I think using third party containers is super useful for is when you need an interactive test environment for something other than what you're running uh on your daily on your laptop so I'm running Mac OS but sometimes I need to test out commands in auntu or Debian or or Alpine and so being able to run this Quick Command and have a interactive shell within that specific Linux flavor is really Priceless like I can test out commands and iterate on them so much more quickly if I'm writing a script that's eventually going to be run in that environment I'll just create a container using one of these commands and then I have an interactive environment where I can test things figure out the exact syntax use the help commands Etc inter inter active test environments for operating systems or for programming runtimes uh we can do something very similar but instead of having a shell in the OS we'll have a a repple in our command line with a particular version of that programming runtime and so if we're if we're testing something that needs python 3111 versus 3110 uh or some minor tweak it's so easy using Docker to get an environment that has the exact versions that we care about for doing that type of test now the final sort of use case that I find super helpful for using thirdparty containers are command line utilities sometimes you just won't have it installed or you don't have the right version and so people have bundled up these utilities and because so many Unix tools are designed to pass data as text from one to another so we can pipe data from one command to the next um we could pass data into these containers and get the output as a stream uh we can use these without actually having them installed on our system so JQ is a utility for working with Json data and so for example uh if I run this command so if I jump back here and we look at the sample data I have here so let's say I have this test.

Json it has key1 value one key2 value to if I run this command it's going to take that test Json data file pass it into this JQ image and perform this command so it's going to find the value located at key1 and combine it with the value located at key2 and so in this case I get value one value two as a string and that's the output and so without having JQ installed on my system I'm still able to use that utility directly with Docker yq is similar to JQ but is for yaml there's also command line interfaces from cloud providers such as the AWS command line or the Google Cloud platform command line uh and you can run those in container images uh you will need to mount in your credentials so by default we know that the container is not going to have access to the host file system and so if your credentials are stored at let's say the AWS directory within your home directory or the config directory within your home directory you'll need to craft the the correct volume the correct bind Mount commands to get those credentials uh into the CLI but once you do that it allows you to use different versions of the C very easily and so if there was a breaking change from one version to the next you can leverage that previous version without having to reinstall and modify things if you've already upgraded now this sounds cool but you might be thinking I don't want to have to type out Docker run blah blah blah with all these bind mounts Etc and so so one way that we can improve the ergonomics of this quite a bit is to use two concepts one is a shell function so we can write a function like this that will take this command and essentially invoke it when we run this function we also can use an alias to do the same thing to to close out this section I wanted to pull up a talk that was given by Jesse frel um I think this was in 2015 so this is quite a few years ago uh Docker was much younger in its maturity however she was an early adopter and employee at Docker and used to run all sorts of things uh within Docker and so I think this is a talk that is worth watching for sure and just sort of shows that you can pretty much run anything in containers giving it a watch is really interesting and inspiring and helps to understand sort of this mindset of I don't necessarily need to install everything on my host system she goes a little further than most with this concept uh but I think it it's still eye opening to see the number and types of things that she's able to run in containers so with that we've reached the end of the first portion of the course in which we learned the history and motivation of Docker about the underlying Technologies we installed and configured Docker such that we're now able to run containers and interact with them and then we saw some use cases for leveraging thirdparty container images for different things throughout our development life cycle at this point we're going to shift gears and start building building our own containerized applications I think it's important to realize that learning Docker in a vacuum doesn't actually provide much value and so instead we want to have some realistic web application with which we can apply this technology in this module I'm going to walk through a sample application that I've built for exactly that purpose it's a minimal three- tier web application and by three tiers we have the front the front end or the client that's built in react we have an an API layer application programming interface layer we have one API built in node.js so that's an interpreted language one built in goang a compiled language we have both of those they're almost identical but I wanted to show two different languages with two different developer tool chains so that we can showcase how Docker handles those slightly differently and then we're going to have a postgres database I've tried to keep each of these components as simple as possible while still being representative of the type of application that you might see in a modern web technology company today the implementation of each of them will be quite minimal but how we configure things like networking and passing credentials will be realistic to an application at a web technology compy at this point I'll jump over to my code editor and walk through this code at a very high level I'll point out though because this course is focused on Docker and containers and not on web application programming I'll keep this relatively high level just to give you an idea of the things that the code is actually doing and highlighting the configuration surface area where Docker is going to interface with them at this point if if you haven't already you'll want to go to the companion repo for this course I'll have a link for it down in the description uh and you can find all the source code that we'll be going through and building out uh and what you'll want to do is here under code uh you can either click here and clone it to your system but even better is if you Fork it because that way you can make modifications commit those modifications and if you have any improvements you'd like to make uh and propose back to the course for others to use you could create a poll request back to my copy of this repo on get Hub and that would be awesome while you're here if you don't mind giving it a star that would be super helpful to help the project grow and while I'm asking for your support you may as well hit the thumbs up button on the video as well and so with that let's jump over to the text editor and see this demo application so I've got the code editor up here uh and we're in module 5 the example web application I'm going to start by running the database just like we did in module 4 I'm going to run the database with a Docker run command I'm going to pass it a postgress password environment variable I'm going to use a volume Mount like we talked about to persist the data outside of the container in this volume so it can live beyond the life cycle of this particular container and then I'm going to publish Port 5432 which is the default Port that postgres runs on and I'm using postgres 15.1 uh and in particular the Alpine variant so this is a make file each of these is called a make Target we can do make run post Gres and now we have that database running in a container I can do the docker PS command container I can do the docker PS command to list out the containers running and we can see here's the container this is the image it was based on uh because I didn't specify a name it generates a random one and we're publishing that Port so that I can connect from my local host at this point let's move and take a look at the node API and get that running as well so it's located within this API node uh subdirectory uh if we look in the package.json file we can see there are just a couple of of of dependencies we have Express which is a package for building rest based apis we have Morgan which helps us configure logging uh for that API a little more easily and then we have the PG package or the postgres package uh which is going to allow us to create a client and connect to the database so what I'm going to do here is make sure I'm running the right version of node so I have NVM installed so I can do NVM LS it'll list out the different options that I have installed on my system I want 19.4 so I'll use nbm use node 194 and then I'll do npm install it's going to grab those dependencies from the package.json and install them in my system in this node modules folder if I wanted to run the application I would do npn run Dev but I also need to pass it one environment variable and that environment variable is the database URL so this is going to give the application the information it needs to connect to that post Chris database running in the container and so again I have the same I have this command laid out here and so I'm just going to do make run API node and now it's started up and listenting on Port 3000 let's just go see how that application behaves if I load it up we see uh at the root path so I haven't specified any path it Returns the current time stamp and I'll show it I'll show you how this is working behind the scenes but it's just querying the database asking for the current time and then returning that and then I have the API node key and value just to show this is the node based API we'll have an an almost identical response from the goang API except this will say goang instead of node for each time we refresh the page uh we're seeing there's a a 200 response that's a successful HTTP response uh for the request and then we have this 404 for the favicon so because I'm loading it from uh from a browser it's also trying to load that little icon that you would see in the top left corner but it doesn't have one so that's the the 404 not found error that we're seeing there now let me just look very briefly at the source code for this application to show you what it's doing we've got two primary files index.js is the main one we're importing some dependencies here at the top we're setting the port number that we're listening on defaulting to Port 3000 uh we're setting up some logging and then here's where we Define that API endpoint where we call our get date time function which I'll show you here in a minute uh and then we populate the uh node we populate the API key within that Json object as well uh in our database do our db. JS file uh is where we have that function uh it's where we're going to load in our connection our credentials so that we can connect to the database instantiate a new uh pool which will be that database client uh and then make the necessary call to the database to get the time that then we we can return on that endpoint and so that's pretty much all there is to this node application like I said I want to keep this as simple as possible while still highlighting the types of networking and configurations that they would we would need in a real world application uh at this point with the node application running we can jump uh we can move on to the the goang application and so for this one it's located at the subdirectory API goang uh here I am within that directory uh to install the dependencies I would do go Mod download uh first I'm going to set the go path just to tell it where to download those modules to to set a workspace um I've got this go workspace subdirectory uh I can export go path and that will point to that subdirectory now if I do go Mod download it's going to look for the dependencies that are in my go mod file and so the two dependencies that I've added here are Jin which is a an API framework work uh and PGX which is a postgress client and so that go Mod download looked in go mod brought those into my workpace okay so now we should have all of our necessary uh dependencies uh and just for reference I'm using uh go19 and so to execute this program I would do go run and then point it to the the main.go file uh that's going to both build the application and run that built binary um again just like the uh node application I either need to pass it an environment variable with the database URL or pass it a file containing those credentials um I have a command in my make file that does just that uh so I can do make run API goang we see it passes that database URL in as an environment VAR variable and then executes our go L and then does gun main.go uh that's running on port 8080 so I can go here and reload this and very similar response to the no l to the node.js one uh we have API goang and then it's grabbing that time stamp from the database each time I refresh it gets an updated time and so just very briefly uh I can take a look and show you the implementation here I'll pull open the the main file uh as you can see we import our dependencies here at the top uh we then have our init function uh within there we're loading in the credentials that we need to to connect to the database once we do that uh we pass those credentials uh to this init DB function which is defined in the database module that I'll pull up here in a minute and then within our main function we have our endpoint defined where we uh call our database. getet time function and take the results of that and we've returned that Json object using Jin I'll just pop open the database module here just to show you the two functions there um we've got our init DB function where we're taking our connection string uh with those credentials uh and initiating a client uh and then I have this get time function which just like in the node implementation is calling select now is now getting those data back and and returning that uh to the caller and so that's really all there is to this application like I said almost identical to node we just want to have both of these different languages so we can show how to manage the different tool chains how we're going to build the docker container images and how we're going to create containers from those images now with the goang API running the final piece of the puzzle is going to be the react client uh so that implementation is here in the client react subdirectory uh once again we can look at the package.json uh we have a handful of dependen we have react and react Dom those are installed by vit when I created this project I used V to do so and then I added axios as well as this react query package which just enabled me to very easily call my apis get those data and feed that data into my react uh components uh so I'll want to once again just make sure I'm using the right version of node I want 19.4 great I'll do an npm uh make sure I'm in the right directory then I'll do npm install this goes off and downloads uh all those dependencies brings them into my my local um node modules folder and then to run it uh I'll just do npm run Dev and by default uh V is going to run that development server on Port 5173 so if I load the page here uh we see the result it's a react app it calls both of those apis gets the data and returns them each time I refresh you can see that the the time stamps are updating and so that's doing exactly what we wanted to do once again I can give you a quick look at the source code just so I can show you what it's doing all the logic pretty much is in this app.jsx file we're pulling in our dependencies at the top I have this function based component called current time that uses the use qu method and axios to make a get call to the uh API that's specified and so this enables me to use the same component for both those apis by passing the API in as a prop I then get the data back and can use that within my component then down here this is the actual content that's going to be displayed I just have which API it's coming from from data. API and then the time stamp that was returned in data.

now while it's still fetching it will just it will have this placeholder up dating text here at the bottom is the content of our app itself uh we wrap it in this query client provider uh that just enables us to use uh the functionality within uh that react query package we've got our static header and then two instances of the current time component uh one where we're passing it the goang path one where we're passing it the node path it is important to call out here that within my V.C config I have these proxies set up such that API goang is sent to Local Host 880 where the goang application is running and then API node is passed to uh Local Host 3000 where the node API is running that's pretty much it for this configuration hopefully that didn't seem too overwhelming uh it's quite complex trying to run all of these Services independently and get them all talking to each other that's kind of the whole point of this course right I want to avoid having to set up each of these Services separately and avoid the complexity of needing to match the right version on your system versus my system and Mac versus Linux versus Windows docker's going to help us simplify this dramatically and so as we progress through the rest of the course we're going to take these components containerize them and then learn how to run those containers how to configure them properly and how to set up a development environment such that we can very easily go from zero to rapidly developing and iterating on our applications all within a containerized setup and what I really tried to do with this application is have each component be the absolute minimum complexity while still giving us that application the networking the configuration be representative of what you would see in a modern-day web technology company and so ideally don't get bogged down with any of the particular applications themselves but just try to understand here is the core of what it's doing we have our client on the front end it's calling those two apis so we need the networking to be set up properly there those two apis instantiating clients and connecting to that database and making a database call so we need to handle those credentials properly as well as the networking between them and just generally we need to have our environment set up for each of these applications to run as desired now that we have our example application we can go through the process of containerizing it now what that means is that we're going to create a specification and then we're going to build from that specification our container image the specification is what is known as a Docker file so it's a text document that's going to have all the commands that we need to create and assemble this image you can think of it almost like a recipe for your application you're going to start with some base layer like an operating system you install your language runtime and any dependencies and tooling associated with it then you would install your application dependencies you might set up your execution environment with certain environment variables and then finally at the very end you're going to run some command that starts your application now this Docker file is paired with what is called a build context we write all of our instructions in the docker file and then the build context is generally a folder or a directory on your local system containing all of your source code it can also be a URL so you can pass uh like a GitHub a public GitHub repo as a build context and we take these two things together and run the docker build command and with those information Docker is able to go off follow the instructions in the docker file and produce a container image I'll call out one additional file that's important to to know about and that's a Docker ignore file within your build context you can include this dot Docker ignore file as the name suggests you can tell Docker to ignore certain files so let's say we've installed the node modules locally we definitely don't want to copy those into the Container image because we're going to install them within the docker file and we could have incompatibilities from the installation on our host system to that within the container and so with this as sort of the highle framework let's go ahead and start writing some Docker files for our example application as we go about writing our Docker files this page within the docker documentation under docs. do.com engine reference Builder lays out all of the valid commands that you can use within a Docker file the format is actually relatively simple if we use a hash that's a comment uh we'll there's a Convention of using all caps for the instruction and then we'll pass arguments after that instruction so for example we can have a comment saying what this step is doing then we can use the Run command which is going to run this during the build phase and then Echo something out if we scroll down here on the right you can see all the different options that you can pass with the Run command as well as all the other commands that are valid we're going to use a number of these and I'll try to call out all the ones that you're going to need to know along the way so let me just jump into the code editor and I will create a Docker file in my API node subdirectory and you might notice that uh I have the 05 module as well as the 06 module I have a whole bunch of Docker files here that show kind of the sequence of events that we're going to go through uh but just for purposes of of demonstrating building it out I'm going to put it in the 05 directory and generally you would want to do this You' want to put your Docker file in the place where your code is living let's think about what that recipe might be for my node.js based application I need to start with some uh Linux distribution and so so I'm let's just use Ubuntu so I'll use the from instruction and then I'll specify an image name so Ubuntu uh we also then are going to need to install the nodejs runtime uh and I can use the APT package manager to do that so I use the Run instruction which is going to run this command uh during the build phase I'll do apt update uh that'll grab all the latest metadata from the package manager so that I can find the correct package and then I'll do run apt install nodejs um and let's just execute the build now just to make sure these steps are working properly I'll navigate to API API API node and then to build this I can just do Docker build with a period I didn't specify a name of a Docker file because it's the default is to use Docker file with no extension I pass my current directory of period as my context uh so it looks like uh it failed because that apt install command uh is interactive and so I need to pass it a Das Das a-y and that'll override that question and automatically install so let's try building it again now it looks like it's installing nodejs and you can see each of our steps we started one of three we we pulled that Ubuntu Public Image then step two of three we ran that run apt update and then in three of three we installed node.js and so if we think back to that union-based file system each command in this Docker file is adding a new layer to that system and so often times we want to minimize the number of layers and think about because Docker is caching Those Behind the Scenes how we can avoid invalidating the cache one such method that we can do to improve this already uh would be to run these on the same line so rather than rather than doing the APT update and then the APT install we can combine them into a single command here and so now those two actions will get combined on that single layer okay great uh so now how are we going to get our application code into this container image and for that we're going to use the copy command so this allows us to copy uh either from our host system uh or later we'll see how to copy from another container image during the build and so I'll copy and I'm just going to copy my entire current directory into the image we're going to improve upon this later but uh but this is just a simple way to do it for now and so with that we'll have our source code within the container then I can run the npm install and that will go off and grab all of my dependencies so let's just make sure this still builds okay so npm was not installed so we also need to install npm okay and now that we had npm installed in the system we're able to execute our npm install command and then the final step that we need to do for the stalker file is to add a command that will be executed at runtime when we create a container the format for this is we'll have an array of strings and so in this case I'll do npm run Dev and that would run my Dev script from the package.json file uh when we started up so I'll build it once more and one thing to note here is that I changed something in my Docker in my Docker context and now when I'm rebuilding it it's executing run npm install and that's because this copy step came before the npm install and so making that modification actually invalidated the cache and so it was much slower to build because we had to run npm install and download all those dependencies we'll address that in a future Improvement now in order to be able to execute this this Docker build C this Docker build command builds it but doesn't actually tag it with any name and so I could run this I could reference it with this shot directly or I can add a-t and I'll call this API node and call it version zero now I can do Docker image list and we can see API node version zero was built a minute ago and look it's almost a gigabyte in size now this Docker file while functional is terrible uh there are so many ways upon which we can improve it and now let's go through and think about what those are first of all starting from a to and then installing uh node and npm while we can do that there are a bunch of official images on dockerhub for language run times so if I go to dockerhub and SE search node it'll pull up the official image and so this is a Docker official image it has that tag it's been downloaded over a billion times and has over 10,000 Stars uh and they maintain all of these different tags for different variants and so if I do this I can remove this install command and instead go straight from node and so let me build that again as version one and so for any runtime any language runtime there's probably an official image that will maintain all the different versions you need and have things preconfigured out of the box so usually you can start from node or from python or from goang and we'll see that when we go to the goang API as well now you'll also notice that I don't have a tag specified here I have the image name but no tag and what that'll do behind the scenes is it just assumes that I mean uh node latest um that's a problem because that node latest tag can change based on the images that that the maintainers of that image have pushed most recently and so rather than tag it with latest it's much better practice to specify an exact version um and so what I'm going to do is look here here at the versions uh Alpine is a Linux distribution is quite small and therefore good for building Docker images because as we're thinking about final image size That Base image that we're building from that's as small as we can get and so it's really important to choose a small base image it's also good from a security perspective to have fewer utilities and dependencies installed from the Geto and so instead of using node with no tag or node latest I'm going to use node 19.6 Alpine and so I'll build that once again and I'll call that one API node 2 and just to show you what a huge difference that makes in terms of image size we can do Docker image list and scroll up here and we see node zero was the iunu based one uh this tag number one was the just node latest version so that's over a gigabyte and then and by using that Alpine base image uh We've cut our image size by a factor of five and we're down to 200 megabytes however you will noticed that each time I'm building it's taking quite a while and the reason for that is I'm copying in my entire Source directory before I do the npm install so I keep invalidating the cache and so the way that we can get around that is that rather than copi the entire Source directory right away uh we can first copy in only the package.json and package lock. Json files and then do npm install and after that copy the remainder of our source code by doing it this way this layer will only be invalidated if we change our dependencies and so I'll call that uh tag number three it's also best practice to specify what's known as a working directory within the image that's the default directory in the file system where commands in the build process are going to be run or the command at the end that we're running is going to be executed when we instantiate the container uh we do that using the work dur instruction and this is going to depend on the convention of your language or framework that you're using uh in this case for node apps I like to use user Source app that's not really going to change anything about our system except that now within the image instead of copying all of our code to the root directory and installing the node modules there it's going to copy things to this path which just keeps things a bit more organized also I have things within my API node directory that I don't want to be copied into this image um there's two things that I can do to help avoid that one I can add this Docker ignore file and in this case I'm specifying the node modules directory so I don't Mo I don't want the node modules that I've installed on my Mac OS system to be copied in because that would cause conflicts and so by putting it in this doer ignore file those will not get copied in I also can specify let's say I only want to copy in my source code directory I don't need to copy in the remainder of this stuff into the image and so I can do that here I'm only copying the source code directory into the Container image let's build it once more we'll call this four now because this is eventually going to be a production container image I don't want to be running npm run Dev here instead I want to run node and then point it to my index.js file and so rather than execute this npm script I will just say no node and then index.js and so that index.js file on my host system is in the S Source subdirectory but because I'm copying from source to my working directory it will be directly it will be in that working directory and I can just call it that way so I'll call that one five and now most of the improvements that we've been making so far either have to do with image size build speed uh Etc another key aspect that we can influence by changing and improving our Docker file is the security of our container and so one very useful technique is to execute as a non-root user and so this nodejs official image happens to have a user defined in it called node and so by by using the user argument and passing it the name node it will use that user now that I've done that though I want to make sure that the files I'm copying in here are owned by that node user so I can use the CH own command node node and now all those source files being copied in will be owned by that user and be accessible by that user so I will copy this tag this as number six now many nodejs applications look for an environment variable called node EnV and so we can use the EnV argument or instruction uh and set node n and we'll set that equal to production if it's set to production the application or the package will behave one way if it is unset or set to anything other than production it will run in a development mode and we don't want that for a production image um so by specifying this environment variable here that can be very useful I'm also going to change from an npm install command uh to an npm CI command and I'm going to add the uh only equals production option and so these two things together the the npm clean install is what CI stands for uh will look for the specific versions from my package. loock file and install exactly those versions so it gives a much more repeatable build and the D- only production option says don't in don't install my Dev dependencies so I have things like just and nodemon in my Dev dependencies within the package.json since I don't want those or need those in my final production image uh I can use this-- only flag uh to achieve that so let me call this number seven and now I know as the person who wrote this app that it is expecting to run either on an environment variable named Port that it loads or it will default to uh Port 3000 I'll add a command called expose uh and just say 3,000 and so this doesn't actually change anything about how the container image behaves or the container this is just documentation for users of this image that it is expecting to run on Port 3000 that way they know to publish that port to The Host or to whatever system they need to so that they can get traffic to the app and now there's one additional place where I can make improvements and that's in this step and this again is going to be to improve our build speeds there's the option to mount a cache location to store information from this step specifically so I'm adding this Mount type cache with a target of this location uh and then I'm setting I'm telling npm to use that location to store its cash dependencies um before calling my npm C call and what that does is if previously if I invalidated the layer 7 it would download all of my dependencies from the internet from npm and then install them now if I've run that once before on my system it will only have to download the new dependencies from the internet it can use that local cache to install the rest of them from that cache avoiding an extra download and just speeding up the process and so I'll call that API node 8 and so with that we have a Docker file that I'm pretty happy with we took that initial naive implementation and improved it along so many dimensions both from repeatability size of the image avoiding invalidating the cache so that F future builds are faster documentation around ports and this the fact that this is a production image using the production command that makes the most sense and using a non-root user within the container now let's take a look at sort of the image sizes as we progress there I'll grip for API node and we can see we started off with those huge images uh when we move to the Alpine based node image as our as our base it cut it by a factor of five uh I believe this small jump was when we when we added the Das Das only production option to our npm install command so there were 30 megab of Dev dependencies that were previously getting installed and then the the change we made there at the end where we set the cach for the npm system uh saved us a couple of megabytes so huge improvements along size security Clarity and this is a Docker file that that I would be happy to use now if you look in the repo on GitHub you won't actually see a Docker file in module 5 that's because I've included them here in module 6 and I've got each of the sequential steps that we took ining it with comments describing the commands that we built up along the way now that we have a container a Docker file and a container image uh for the nodejs API that we're happy with let's go ahead and shift gears into the goang API and build one out for that now similarly to with node I could start from auntu but instead I'm going to use the official goang image and because we're doing it the naive way we're not going to tag it we're going to use that latest uh we're going to use that latest one and then we'll come back and improve just to see how much to see how much that saves in our final image size but yeah we can start from that golang image and so I can build it with just that from command this image would be identical to the the Upstream base image at that point like with node we want to set a working directory uh in this case I'm just going to set it as app we need to get our source code into the image so again we use the copy command let's just copy it all we're going for the the simple naive implementation first and then we're going to improve upon it to get the dependencies installed uh we'll use the Run command with go Mod download so that will look at our go mod file and download those dependencies and finally we can set our Command and for this I'll use go run and then point it at my main.go file uh so this should be a a functional Docker file and there we go let's do a a Docker image LS and we'll GP for API goang okay again over 900 megabytes over the set of improvements that we're going to make we'll improve that quite a bit I will add uh tag zero to that one just so we can remember it we never want to use an untagged or latest image and so we'll add the 1.19 Aline tag and then we can rebuild with that and so already just that has cut our image size in half which is great but we're not going to stop there just like how with node we didn't want to run our npm run Dev script because that's a development specific thing uh here go run actually builds and runs our application and rather than doing that we would much prefer to build our application in the docker file and then at runtime we can just call that built binding and so instead add a run command I will do go build- o API go langang that's going to create a binary called API goang from our main.go file and then rather than use this as our Command we can just run API go langang we'll call this number two this change means that every time we start up our container application we no longer need to build it because it's already built so we're just shaving off time that it would take in that container bootup process at this point I'll also point out the docker ignore file that I'm using for this uh project I've added the make file the readme and uh any Docker file that I create to the docker ignore those are files that are that do not need to be in the image and so this prevents uh when I make that copy command from invalidating the cache uh each time I change the docker file each time I add something to the readme ETC now similar to uh node we don't want to invalidate the cache here before we download the dependencies because then we'll end up downloading those dependencies way more often than we need to and so I'm going to move this copy command after and instead we can just copy the files that contain the references to our dependencies and so that's the go mod file uh and the go.

suum file and we'll copy those into the uh current directory I'll call that number three with that change we can make changes to our source code without invalidating the cache before downloading dependencies and so each build would be much quicker and now at this point we're going to add a new technique that I haven't shown yet and that's something called multi-stage builds and these can be super powerful because it allows us to have within a single Docker file multiple individual stages that are building separate container images now for a language like goang that is compiled this is super helpful because we need a lot of dependencies in the build stage where we're using them to build our code however the end result is this binary that can have all of its dependencies statically linked we don't need any of that golang tool chain in the final image I'm going to label this stage as build and then I can add a new stage down here and in this case I can use a special image called scratch and this is a official image that we can look at here let me just go to Docker Hub uh it's a Docker official image it really has nothing in it the only thing that it has in it is a valid root file system it's effectively a noop that gives us a place to start from that we can copy our binary into and so now since we're starting from that scratch image we can use a copy command but in this time instead of copying from the host we're going to copy with a D- from equals build and so now this is saying copy from this container image that was built in that first stage and now we want to copy the slapp Slash API goang and we need to use SL apppp because that was our working directory that where that binary ended up uh we want to copy that to I'll just copy it to API goang and so it will be at the uh root path and then we can call it from our uh we can call that when we instantiate a container from this image uh however that wouldn't work as is because this build command didn't statically compile all those dependencies in so I need to modify that build command and I can do so like this so I've added these uh LD Flags to change the link mode to external uh and set the compilation to static I've also added this netgo tag I found that I needed to add that to make my networking within the image work properly however by doing that I now get this compiled binary that has all of those dependencies built in and I can run it from the scratch image so if I do this and build it I'll call this one four o but when I built it uh I ran into an issue where my build stage didn't have the compiler required to do that static linking uh and so one way that I can fix this is to use Buster which is a dban based distribution my Alpine image while it was smaller that doesn't matter as much anymore because this build stage is separate from my Deployable stage I'm going to end up using this image to deploy and so even if this is slightly larger in my build stage uh my Deployable stage should still be quite a bit smaller so let me try building that again and so that can be a trick when you're using multi-stage Docker files is you care a little bit less about the size of your intermediate stages and so you can use an image that has more build tools available uh or slightly easier to work with and so in this case we're using that deian based image in the build stage and then we're still getting the benefits of that tiny scratch uh stage for the output so if I look at the image sizes here we've now jumped from 500 megabytes to that statically compiled one is only 16.8 megabytes so huge benefit by using that multi-stage build compiling to a static binary and then copying that into a separate stage a couple things I want to add to set up my production environment uh and that is environment variable for Jin which is the API framework that I'm using and so I'll do gin mode and set that to release so similar to node end production Jin behaves differently whether or not this environment variable is set and then I know that this app is going to run on port 8080 and so I'll set expose 880 there as documentation uh for the end user I'll build this as tag number five and then the final Improvement that I'm going to make is similar to the cache Mount that I use for npm but we can do the same thing for our go Mod download command uh and so I can add a cache Mount here um this slgo package mod is and the root cach go build are the two directories where uh those dependencies get cached as they're being downloaded and installed and so adding this can just speed up subsequent builds accordingly and so as my final image I'll call this six I guess one final thing that we can do from a security perspective like our node based API we would prefer to use a non-root user within the side of the container and so to do that we're going to need to set up that user uh in our build stage and then copy a couple of files into our release stage that will allow us to use it I can do that with the uh run user ad command uh and I'll do user ID 1000 and I'm calling it nonroot then in my release stage I will need to copy the password file from my build stage so I can do copy from build SL Etsy password that will copy that in and then I can do user non root and that will use that user we'll call that number seven now in this case whether that was necessary or not is a little bit questionable uh because we only have our goang binary that's the only thing pretty much in that scratch image the tax surface area is quite small and so maybe it's over the top to add this nonroot user but I just wanted to highlight the fact that we can do it and it provides one additional layer of protection uh to our sort of Defense in depth approach uh when it comes to security and so with that we have a Docker file for our goang based API that I'm relatively happy with again we've pinned our our base image uh we've used a non-root user we're avoiding invalidating the cach when possible we're leveraging this Cash Mount to speed up future builds uh we're compiling to a static binary and leveraging a multi-stage Docker file to really take advantage of a much smaller image in the end uh and we're setting up our final environment with things like this environment variable um and the expose command for the port that we plan to run on and so with this we can create a container for our API that's both small and secure just like before if you take a look at the repo you won't find a Docker file in the module 5 instead I have created a number of Docker files here in module 6 that walk through all the steps that we just went through and have commentary around why we're doing the different things that we're doing so we've got two of our three applications containerized we have the docker files created the final one that remains is our uh react client so let me navigate to that and I will create a Docker [Applause] file uh and with this one the development mode for a v based react app uh is going to be a node JS server serving those files and so you'll see that our Docker file looks pretty much identical to the one that we had uh for the node based API so this would be a valid Docker file for that again let me build it so I'll do Docker build- T we'll call it client react zero we'll build that as always before that even finishes building I'm going to add a pinned version of that base image for repeatability and image size uh in this case we're not going to we're going to end up using a multi-stage build as well so I'm going to use the Adian based base image just cuz it's a little easier to work with than Alpine sometimes and I'm not as worried about it since it's not going to end up in the final image so I'll tag version one as the one with the the 194 Bullseye tag similarly our Docker ignore file is going to have the node modules directory within it because we don't want to copy those into our our image two improvements that we saw with the node API that we're going to make here as well is we don't want to invalidate the cache by copying in our source every time instead we're going to copy in the package.json and package lock. Json and then also let's set a working directory as sluser SL Source slapp I'll tag that as version two uh changes like that don't actually modify the size that much of the final image but it's just useful to see kind of the progression uh I'm going to use the same cache Mount approach that we used before to speed up future installs I will use the npmc command here I'm not going to use the Das only production flag because in this case it's our build stage where we're going to generate some static files rather than the final image that we're going to deploy a container with and so you'll see here our Command is still currently npm runev that's going to run a development server that's not what we want what we actually want to do is run npm run build which is going to generate a set of static files HTML JavaScript and CSS that then we're going to be able to serve so this will generate those files I need to name my first stage because we're using that multi-stage build approach so this will be as build we're going to add a new stage here at the bottom uh and we can use from engine X and let's see what version of enginex we want to use uh let's use 1.22 D [Applause] Alpine we're not going to need a command we're going to use the default engine X command um but in order to make this work we need to get the files that we built in line 13 and copy them over into this final release stage and so to do that uh we'll use the copy command we'll specify from build that's saying from that top build stage look at the path user Source app so we have our work directory user Source app and then the npm Run build command is going to place those files in a subdirectory called dist for distribution uh and then this path is the location this is the convention for where you would store uh your website files if you're going to serve them with enginex and so that's what we're saying there we need one more copy though and that is to copy in our enginex config and so our enginex config is how we're going to set up uh engine X to serve our site the way we want it to and so this is being copied from the host I have it here let me just walk through this real quick we've got a few endpoints set up we've got a ping endpoint which just returns 200 um pong directly I also have these two additional endpoints set up as proxies so if you go to/ apiol Lang that gets proxied to this address um and if you go to SL API node it gets proxy to this address and we're going to set up Docker when we run it uh to host those two apis at those paths uh within the network that we set up and then finally the root path if you don't hit any of those you'll end up here it's going to serve our react app files from that directory where I copied them in the final thing that I'll do is just to once again have an expose command this is just going to be an HTTP server so we're exposing Port 80 uh that's where it's expected to listen all right uh let me build this as three um and actually this engine X image the official one runs as root and so one way to improve that as always is to run as a non-root user and so there is a uh another image that I found which should be pretty much identical to that one um let me just search for it here this engine X Inc engine x unprivileged uh is a very popular one it has over 100 million poles uh it's very similar to the official enginex image with a few minor tweaks around uh the default settings including the user that it is run as and so in this case I'm going to swap out uh from that image to this unprivileged image and rebuild it once more as tag number four and one of the key differences when you're running unprivileged is that you can no longer bind to Port 80 uh you can no longer bind to any port below 1024 I believe and so rather than expose 80 the default here is going to be to run on port 8080 uh and so I'll change that and with that we can build it once more and let's just look at the image sizes as we progress so we can do instead of that we're looking for client react uh we started with a massive 1.16 GB image uh then when we switched to the engine X image it dropped to 23 megabytes uh the unprivileged engine X image isn't quite as optimized I guess so it's 76 megabytes but I think the Improvement of running as a non-root user is worth it for for that extra image size as with the other projects uh within the module 6 I have a series of Docker files listed out here that show the improvements along the way along with commentary and so feel free to look at them and see the differences between each step why we're making the changes that we did now going through that process hopefully it was educational it might have felt a little bit repetitive but that's kind of the point we want to find a set of general principles and practices that we can apply to any Docker file for any container image that we're building and the general approach that I think about is first we want to make it work that's kind of the fundamental thing that needs to happen that's that naive base Docker file that just has a few lines gets it running then we want to think about two things we want to how do we make this more secure how do we make it faster to build so it's easy to work with I've laid out a legend here that talks about the three types of improvements that we're making whether it's impacting the security of the image uh the speed with which we can build build it or the clarity of the image how clear it is for future developers who are coming along and reading it and so for each of these techniques I've listed out with the Emojis so that you can see what they're doing we pinned specific versions within our build this can impact security speed and Clarity the base image is a huge one right if we pin a specific small secure base image that's a great starting point we can either use a tag like I was using 1.19 Alpine you can also use the hash of the image itself to use it or a combination of the two so you can have the tag and then an at signed and then shot 256 and then the hash that's actually kind of The Best of Both Worlds where you have a human readable version as well as that immutable hash that will correspond to a specific version that can't change you also want to pin specific system dependency so if we were installing something from apt or a package manager we would want to install a specific version there so that it doesn't change on us uh that can help with security as well as Clarity so you know exactly what is going into the image uh and then application dependencies so within our package lock Json file uh within our go mod or go Su files we want to install the exact depend every time so that we're not accidentally changing things from one build to the next I mentioned this in the pinning of specific versions but it's worth calling out again you want to think about what's the smallest and most secure base image that I can use that will still be nice to work with and I can install my app into it that's going to help both from a security perspective as well as a speed perspective because those smaller images are just easier to move around whenever we're adding a command to a Docker file we're adding layers and we want to think about how do we protect that layer cache such that future builds can leverage as much cash content as possible the general principle here is to order your commands by how often they're going to change so that base image is very infrequently going to change we want to install our dependencies before we copy in our source code so that we don't bust the cache there we want to use that cash Mount technique so that it can store downloaded dependencies locally and reuse them in the future this next one is a a technique that I didn't actually highlight in that process it's a relatively new feature it's an option to the copy in instruction uh and by using copy-- link it creates a new separate layer that's not tied to the previous layers and this is super useful in the multi-stage build environment if I'm copying from the previous stage I can use this such that if I change my base image on my second stage that won't invalidate this the cache because this copy. link when I'm copying from that previous stage it ties it to that layer stack instead and then the final step is to combine steps that are always going to be executed together so that we have fewer total layers in our image we also wherever possible want to be explicit within our Docker files just for clarity sake uh we want to do things like set that working directory so we know where within the file system we're operating we want to indicate the standard Port that we're going to be listening on with that expose command for clarity and we want to set the environment variables how we want them with the end instruction so it'll be automatically set when we run that container we also want to avoid unnecessary files in the container image we can do that in two ways we can add things to the docker ignore file so that they don't get copied in we can also when we issue our copy command copy specific files or subdirectories rather than just copying in the whole build context for security purposes it's always better to use a non-root user so we saw how to do that in a few different cases uh we also want to only install production dependencies so if there's certain things you only need in your development environment there's no need to build those into the Container image uh we always want to avoid leaking sensitive information uh and so good example of this that I'll show here in a minute is if you have a private repository that you need to log into you could have private npm packages maybe you're building a python app and you have and you have a private python registry we'll need to pass those credentials in a specific way so they don't get built into a layer within our container image and then the final one here is just to leverage multi-stage builds where it makes sense we saw how with the goang API we were able to cut our image from 500 megabytes to just 16 with the react client we were able to go from that node based build environment to that optimized engine X image at the end it saved quite a bit of size and just makes a lot more sense for serving those static files that are generated as I built out those images I often chose an Alpine based base image I was doing that because of the size however size is not the only consideration when choosing a base image for your container it is an important consideration because the size of that base image is kind of the lower bound on the size of your final built image however you also want to think about how well that particular base image supports the language that you're running in in this case it may have not been the best choice for a node.js application if you look at the official nodejs Docker file page you'll see that Alpine is still considered experimental uh and that's because it uses a different variant of the standard C library uh and so there can be subtle bugs associated with that if your code or the libraries that you're depending on are making assumptions about the implementation details of that core Library you'll also want to think about the ergonomics of the image how easy is it to work with if you need to install additional packages does it have utilities built in for debugging Etc and then finally the security of the image is another huge consideration things like the number of cve built in at what level uh and the attack surface area if someone is trying to compromise the container on the right hand side here I've listed out some sample images that you could use for a node.js application the first one is the default latest tag which is currently pointing to the 19.8 Bullseye image and Bullseye is a variant of of of dban that one is nearly a gigabyte it has five critical vul vulnerabilities in it generally you wouldn't want to use that however it is quite easy to work with you can use the app package manager to easily install lots of different Packages Etc the next one down there is the slim variant of that so it's still using that Debian based image it is cut the size by a factor of four eliminated all all of those critical high and medium vulnerabilities it still does have some lowlevel vulnerabilities cves however it's a much better choice than the fullsize image for that reason however because it's slim it won't have a lot of utilities built in out of the box and so you may need to install those if you need them at either build time or at runtime uh the node Alpine image is the one that I was using in some of the examples it doesn't have any cves built in it's quite small however as I pointed out and I actually learned this after filming that section it's still considered experimental by node and so if you go to the repo you'll notice that I'm using that slim based Bullseye image for the the base image there there's two additional images here at the bottom that I've listed out the first one is what is called Dro list that was created by Google as a security focused image that only contains the language runtime that you need it doesn't have any additional things like a package manager uh shells built in ETC and so it is quite small and reasonably good from a security perspective however one potential downside is that it only supports LTS long-term support variants of node and so if you want to run on a different version of node that's not possible it also if you need to install anything outside of node there's no package manager so that could be more difficult to do the final image that I've listed there at the bottom uh is from chainu guard and chainu guard is a security focused Cloud native company they have started producing and maintaining uh these BAS images for various languages and run times it is quite small as you can see it's the smallest one listed here it has no known cves in it and it's designed specifically with security uh and software Providence where the source of any dependencies are coming from because it is more new uh and it is really locked down it may be slightly more difficult to work with in terms of the types of packages that are supported and included if you have particular needs for your application and so I think as a general purpose image that's easy to work with the dbn based variants the slim variant in particular uh is quite a good choice and if you're really looking to lock down your eventual containers something like the the chain guard based image uh could be a good choice through the process of building out those Docker files for the demo application I highlighted a number of different instructions however I didn't cover them all there's a couple of additional features that I want to call out here included this Docker file. sample in the repo as well um the first one is paral directives so at the very top of your Docker file there's two things you can do one you can specify a specific version of the docker file syntax that's necessary if you're using some of the newer features that are not enabled by default so that copy-- link that I that I talked about a minute ago um that I I think is in Docker file version 1.5 so if you wanted to use that you would need to have this directive at the top of your Docker file uh you can also set an escape character the default Escape character is going to be that backslash but if you wanted it to be something else you could set it accordingly the next feature that I'll call out is is label and this is super useful for including important metadata within the image manifest for example there on line 14 I've included myself as the author of this Docker file you can add any arbitrary key and value pair there are a number of standard specific labels under org open containers image that people use including things like authors version Etc so I would suggest taking a look at those and deciding which ones make sense for your images I also mentioned that sometimes you want to combine multiple commands into a single step so they're combined within a single container layer one nice feature for doing that is what's called the her do syntax and if we see they're on line 20 we have run and then the less than less than eof apt update apt install eof that allows us to write multiple commands across multiple lines however they get interpreted as being a single line within the docker file and so we'll execute those two things in turn but they'll be stored within a single layer within the container container image in order to avoid sensitive data from leaking into our our image at build time we will want to use the secret Mount type so on line 30 there I have an example of that where we're mounting in a secret so this in this case it is a on our local system it's a file called secret.

text and we can mount that into the file system of the container at/ container secret. text uh and that will only exist at build time it will not get included in the final artifact that is an important technique to know if you need to use credentials within your build context now so far we've been using the CMD or command option and that specifies the command that gets run when you create a container from your image there's an additional instruction called entry point which is slightly different and a bit nuanced as to how these two interact with each other it can be a little confusing as to which of these get gets run in which case if you're building your own application and it's always going to be run with the same set of arguments you want to just place all of those in the command like we've been doing however if you're building a utility where you can pass additional arguments to The Container at runtime you would want to put the executable in the entry point and then allow people to append arguments via the command at runtime when we wanted to get files from our host system or from a an earlier build stage into our container image we were always using the copy command there's a very similar command called add which is a little more ambiguous and so Docker generally prefers using the copy command you may see Docker files though that use ad and just know that they perform almost the same way but there are a little bit of nuance in the Syntax for very specific edge cases so just know that that exists and then the final thing to call out is there's a technology called build X which is built into the docker CLI which allows us to do a number of things but one of the very important things is to build and run multi-architecture images so if I'm on an x86 based system but you're on An Arm based system or I want to deploy to An Arm based system I can use build X to build versions of of my image that are compatible with either of those now I'm not going to go into detail in this course however I do have a video that does a deep dive on buildex and building and using multi-architecture images that I'll link to in that card up there I will just pull up this Docker file. sample and just interact with it a little bit though so within module 6 I've got this Docker file. sample uh and then in my make file I have a couple of commands that I want to show you here uh the first thing that I'm to do is make build sample because it's using that secret Mount that's a build kit feature that you have to enable and so by specifying the environment variable Docker build kit equals 1 I'm able to use that I'm also setting this base image tag as a build argument so one important thing to call out is that within a Docker file I can use this ARG instruction that will be an environment variable available at build time but not available at runtime and so in this case I'm setting my base image tag it was going to default to 194 but I was able to override that with this base image tag here of 193 in my build command however even though they're not available at runtime they still exist as artifacts in the image metadata that's why we can't pass credentials using this buildar uh approach instead we want to use the secret Mount approach now you'll notice that I have both an entry point and a command here uh and so I'm going to run this image and let you with a few different options and let you see how that behaves and so to do that I have this run sample entry point command so I'll do make which goes off and builds our image and then it's going to run it in a number of different configurations if we run the image with no arguments so if we just do Docker run and then the name of our image uh what do we get out we get out hey Team wave entry point plus command so with no arguments we get we Echo both this string and then we get this string as well so both of those get passed to the echo command if we run with an argument so that would be like Docker run sample plus argument now we we still get hey Team Wave entry point but instead of plus command we get plus argument so adding that argument onto my Docker run command overrode the command that was specified in my Docker file we also can override the entry point at runtime and so here if we do Docker run-- entrypoint Echo sample that overrides the entry point and it ignores our Command and so that comes back with an empty string because we ran the echo command with no arguments and then the final case is if we override the entry point and we pass additional arguments uh it's going to do it's going to override the entry point ignore the command and use the arguments that are passed at runtime and so with this command we end up with hey team overwritten entry point plus arguments hopefully that description shows you how those two things interact like I said if you're building your own application that's not going to be called with different arguments each time put everything within that Command Block it's just simpler and we don't have to worry about the entry point you're generally only going to use the entry point if you're building out a utility that's expecting additional arguments at runtime to be passed into that container now I do have a make Target here to also demonstrate Building multi-architecture images so I'll to make build multiarch uh in this case I have a a repo on Docker Hub called Sid multi Sid Palace multi-arch test so I can show you that and this is going to with this D- platform tag it's going to build one image for amd64 one for arm 64 and one for arm V7 which is a 32-bit uh arm architecture it's going to build that based on the docker file.

sample and then it's going to push it to dockerhub so if we go here it is still building once that finishes building and pushing uh we would see an updated tag I last pushed it 20 20 days ago but if I update this here in a minute it will update to now we can see it's working on those different architectures in par parallel and so behind the scenes build X and buildkit are using uh a emulation technology called Kimu qmu to enable us to build the an architecture other than the one that we're actually running on this this topic is becoming more and more important as the number of developers on arm systems with apple silicon is increasing and the fact that arm-based servers in the cloud are generally quite a bit cheaper than their x86 counterpart so being able to build for whatever architecture you want can increase the likelihood that your team is all compatible and allow you to take advantage of some cost savings with your eventual deployment so it looks like it is finished so if I refresh this we should see that we just pushed a few seconds ago for all three of these architectures uh with that test image and that concludes our Deep dive on writing and optimizing Docker files hopefully you'll be able to take that knowledge and now apply it to any future application that you'll need to containerize in this portion we focused on building out those production container images in later modules we're going to think a little bit more about how do we modify these to make them easier to work with from a development perspective so that we can have both our optimized production image as well as a development and debug or test image that we can work with on a day-to-day basis so far in this course we've been building all of our container image locally but if you're working on a team or you need to deploy those containers somewhere you're going to need to start using a registry a container registry is a repository or a collection of repositories used to store and access container images so these are generally stored somewhere in the cloud and they enable us to take those images that we've built either locally or within a continuous integration server uh push them to this registry and then other team members will be able to pull them we can pull them into whatever deployment environment we're using repos can be either public or private so when you go on dockerhub and search for those public images you'll find those but you can also set them up such that they require authentication there's many different container Registries that are available dockerhub is a popular one that is provided by Docker the company GitHub has its own container registry Google Amazon Azure they all have container Registries as well as I mentioned you can have public or private container Registries for a public registry you don't need to authenticate in order to pull however in order to push to that registry you would need to authenticate and for a private registry you'll need to be authenticated for both pushing and pulling Docker desktop has what's known as a credential Helper and using this when you log into one of these container Registries it's able to access system resources such as on Mac OS it stores the necessary credentials within your system keychain so that they can remain secure on your local system additionally some Registries May leverage their own tooling to authenticate for examp example the Google container registry uses the g-cloud command line utility to authenticate your gcp user to that registry and works with the docker client to provide access now I want to demo uh building and pushing an image to dockerhub and the GitHub registry just to Showcase how we would do that I'm using the most simple Docker file you could imagine here it's just going to have a base image from scratch I'm putting that into a Docker file building my scratch image from that and then I'm deleting that doer file just because I don't want to store an extra file in the repo so if I do make build we've now built that container image and it exists locally on my system in order to push to a remote repo you have to do two things one you need to be authenticated to that repo and two you need to tag an image with a tag corresponding to that repo and so if I go to dockerhub I've already created this repo called my scratch image if we needed to create a new repo you would click the create repository button uh fill out the name add a description choose if it's public or private and then click the create button in this case I've already created this and so the tag that we need to use in order to push to this repo is going to be Sid Palace which is my dockerhub username the name of that image my scratch image and then we can append additional tags to the end in order to authenticate to dockerhub we would use the docker login command I've already authenticated so it used those existing credentials but it would ask you for your dockerhub username and password in order to log in if you hadn't done so before so in order to take this image that I just built and push it to dockerhub I need to retag it with a tag associated with that repo so in this case it was sidp Palace my scratch image uniquely identifies that repo for Docker Hub you don't need to include a prefix you can just use the the username and repo directly and it will assume that it's dockerhub if you don't have a registry prefix in front of it we'll see that's a little different for the GitHub one but in this case I'm using the docker tag command to take my original tag of mys scratch image that I applied up here uh add an additional tag of Sid Palace my scratch image and because I don't have a colon with a tag uh version it's just going to apply the latest tag to to to it I can then push it to dockerhub and that will show up however generally we never want to use the latest tag and I'm going to talk about tagging best practices here in a bit but it's much better to apply a specific tag to this so I'm going to once again take my original tag of my scratch image apply a new tag which corresponds to my dockerhub repo and in this case I'm adding the tag abc123 as just an example tag I can call a Docker push on that tag and it'll get pushed to to dockerhub so let me just call that make push dockerhub uh it tags my images with the ne necessary things and then pushes both of those to dockerhub I can now go to dockerhub and if I refresh the page we can see both that latest tag when I didn't specify anything as well as my abc123 tag were both pushed uh just a few seconds ago so that's dockerhub let's now go through the process of authenticating and pushing to GitHub container registry the process looks quite similar but we do need to follow the instructions um at this link and so here's a walk through of how to work with the container registry in particular I want to look at authenticating to The Container registry they give some instructions on how to do this within a GitHub actions workflow um I'm going to use this authenticating with a personal token option and so the way that we do this the set of minimal permissions that we need can be created using uh this URL that they provide zoom in just a bit uh I'll say Docker course creds it has the Right Packages permission now I'll click generate token don't worry I will delete this token once I finish recording so I'm going to export this as my uh container registry personal access token great and then there was a login command here yeah so I'm going to Echo that token out and pass it to the docker login uh command specifying that I want to log into the GitHub container registry. registry so Echo that looks like our login succeeded and now just like with dockerhub I'm going to retag my image using uh this repo name and so in this case I have the same repo name as I had on dockerhub but now I'm prefixing it with gc. which tells it to use GitHub container registry again if you don't specify tag it will default the latest and then down below I'm tagging it with abc123 and pushing both of those to GitHub so I'll do make push GitHub packages it tagged both of them pushed them to GitHub container registry and now I can navigate to GitHub container registry Sid Palace my scratch image uh and we can see we have a latest tag and an abc123 tag now let me just go delete those personal access tokens so I don't forget and that's under settings developer settings personal access tokens tokens revoke all so each registry is going to have its own instructions for logging in so find the instructions from the from the registry itself and then follow those and that'll work with the docker client to store those authentication credentials on your system now I talked about when I was tagging and pushing those images how if you don't specify a tag it defaults the latest and that's generally a bad practice you want to have descriptive tags that tell you what the image actually is doing it's also important to note the same image can have many tags and so for example if we go into dockerhub when I last checked on January 30th all five of these tags auntu latest auntu 2204 Jammy -22 1130 Jammy and this specific shot 256 hash all point to the exact same image and so when we're tagging our images we can put multiple tags that we think will be useful to the end end end user now ideally you want to treat any tag except for maybe a temporary one that you're using for development as immutable and by that I mean you shouldn't create a tag push it to dockerhub or or whatever you're using for your your registry modify the image rebuild and tag and then push and overwrite that can be a nightmare if someone else is using that image and they could accidentally get the updated version and cause issues within their process so you always want to treat tags as immutable where possible the one exception there is if you're doing something locally and building it over and over and running it locally you might not need to change the tag every time because you're you know the fact that you're changing it every time now usually your images are going to be built within some continuous integration system hopefully that's automated and often times I'll see people use some combination of a time stamp of when it was built so maybe like we see up above that 22 1130 uh it was like likely built on November 30th uh you might put in a build ID from your continuous integration system so maybe the GitHub action the ID of that GitHub action so if you need to go look and see when it was built you can see the process that happened uh the commit hash of the source code so let's say I'm building an image based on the latest commit to my main branch I might take that that hash from that Branch uh of the latest commit so that I can very easily find where in the code base uh that build is coming from uh and then also often times I'll see people use a sver release version so like 1.1.0 or whatever the the latest release version is these can be combined in various forms uh when I set up an automated workflow with GitHub actions in the developer experience portion I'll tag the images with a bunch of with combinations of a bunch of these also as we saw when we pushed to a registry other than dockerhub we had to prefix the name of that registry at the front end of our tag now that we've built our container images and learned how to push them to various container Registries we can move on to running them and configuring them to all talk together within Docker up until this point we've used the docker run command a number of times to run containers from third party images we've seen this example with postgres a number of times however there's another way to run Docker containers that is built into Docker and that's called Docker compose behind the scenes they do exactly the same thing but Docker compose allows you to specify all of your application configuration within a gaml file and it makes dealing with the container life cycle much much easier these two images here on the left hand side shows the docker run variant the right hand side does exactly the same thing but using that compos file instead if you're just running a one-off container I think using Docker run is fine but for our situation where we have multiple services that need to talk to each other Docker compose is much more intuitive and easier to work with I'm going to show how to do both for our setup but in most situations you'll be using Docker compose if you have an application with multiple containerized Services I'll also point out this GitHub repo called composer eyesee which you can paste in a Docker run command and it will generate the equivalent Docker compose file and so if you have a command that you've been using and you want to convert that to Docker compose that can be a very easy way to do that now there's a ton of different options and flags that you can use at runtime with Docker I've tried to pull out here the most common ones that I use on a day-to-day basis and let's go through and test these out and showcase what they do the first one is that Das D which stands for detached which allows us to run a container in the background so as an example I can run Docker run auntu sleep five and it's going to take over my shell until that sleep command exits and then it will return so after five seconds it's done however if I add A- D to that now it gives me the ID of the container and runs in the background I can see it by doing Docker PS and here we see that abun 2 container uh running my command sleep 99 in the background the next one is the D- entry point option where we can override the entry point that was defined in the docker file an example of this by default the auntu image doesn't have an entry point and uses bash as its command we can actually use the entry point like this to say roer run entry point override that with Echo and then we'll pass the argument hello so when I run this it should Echo hello back to the terminal there we go the third one there --v or- e or --v file is to set environment variables at runtime so we could do things like pass in database credentials or configure anything else in the environment that we need to Showcase that one I can do Docker run-- EnV my n equals hello auntu and then the command I'm going to run is print end which will print out all the environment variables in the environment you can see it has these three environment variables including the one that I set at runtime the init flag uh is an interesting one some programs are not designed to be run as process ID number one which is what happens by default if you run them in a container and so if you pass the init flag with your Docker run command instead Docker will run its own initialization script and that will spawn your process as a subprocess so for example I can run Docker Ubuntu PS and that PS command gets executed as process ID number one however if I pass the init flag and run the same thing we see that this Docker init is process id1 and my PS command is process ID 7 this is helpful if your application is managing subprocesses itself passing the init flag can make it easier to handle forwarding along termination signals from Docker to The Container the interactive flag or - I and the TTY flag DT allow us to have an interactive TTY session inside the container so if I just run Docker run Ubuntu it will call bash which will exit immediately however if I add the dash it doer run-it Ubuntu uh the - I will make sure that standard in remains open and the- T will give me a TTY teletype or shell session with inside the container and now I have the shell running in the container we've seen the D- Mount and D- volume options used quite a bit when we need to persist data outside of the container layer in a volume so I'm not going to demo that one now the name option gives us the opport to provide a specific name for a container otherwise Docker will assign a random name when the container started so I can do uh Docker run I'm going to use the DD flag so this runs in the background and doesn't take over my terminal I'll name the container my container and then pass it the Sleep 99 command ah and this actually shows one downside of using named containers is that you can't have two containers with the same name on your system so before I could run this I would need to remove that container now I should be able to run this and now we have that container in the background if I do a doctor PS uh we can see under names my container this can be useful if you want to be able to quickly grab the logs from the the the container you can now address it with the name rather than the ID that that container doesn't have any logs because it's just the Sleep command but usually I just let Docker manage my naming for me and do it randomly the network or net option allows us to connect to a specific Docker Network so if I do Docker Network LS we can see the different networks on my system I'm going to add a new one called my network with Docker Network create my network now we can see that new my network was just created and now I can create a container that attaches to it and if I inspect that container with the ID and grap for Network we can see it's running in my network specifying a network like this allows us to create isolated networks for any for our applications rather than just letting all of the docker containers end up on docker's default Network in such a way that they would be able to communicate with each other the platform option allows us to specify which architecture we want to run our container image even though I'm on a amd64 system I can run it with this Linux arm64 V8 option it went off and pulled the arm 64 version and then I'm running this D package print architecture command and we can see that it output arm 64 if I do the same thing but specify Linux amd64 we get the amd64 result we've already seen the publish or- P option used quite a bit to connect a port from our host system to that of the container the Das Das restart option there's a few different options you can have always unless stopped or Never So if I specify to restart this container unless stopped Docker is going to see if it exits and unless I told it to stop it will continue to try to restart it so in this case that bash command the default for auntu is going to exit docker's going to see that and restart it over and over so now I can do watch Docker PS and we'll see this image is exiting and restarting exiting and restarting and then lastly the-- RM flag tells Docker that if the container process exit it should remove that container rather than leave it on the system as a sto container this can help a leave at that naming conflict that we ran into before where the two containers had the same name if we had used the D- RM flag when the previous container had exited Docker would have clean that up to demonstrate this I can run one Docker container uh named this one will be there that's going to exit however it should still remain in the system as a stop container I'll now run another one with the-- RM flag called this one will be gone and now if I do a Docker container list- a the d a is necessary because otherwise it would only show running containers and I grep for this one that should find both of them if they existed however we only see this one will be there in the listing there are way more options than this including this set which I think you should be familiar with you may not use these on a day-to-day basis the first of which is capabilities add and capabilities drop this allows you to specify which Linux capabilities should be accessible from the container so this is a security feature where you can what is allowable within the context of that container cgroup parent allows you to specify which cgroup ID your container should be associated with if we think back to how we can use cgroups to Monitor and limit the resources that the container has access to this is one way to specify that directly CPU shares is another cgroup related option where you can specify what percentage of the CPU Cycles this container should be able to access the CPU set option allows you to specify which core specifically so if you're fine-tuning performance and want to pin a process to only execute on a specific CPU core you can do so these next two are related to the different devices that the container should have access to and at what throughput and bandwidth if we wanted to be able to access gpus with our container we would want to use the GPU flag we can use these health related options to specify a health check that Docker will use to periodically ping our container and make sure it is is healthy like our CPU option we can use this memory option to specify how much memory the container process should have access to if it goes over that limit Docker will kill it we can use these PID related options to specify how many subprocesses our container should be allowed to manage the privileged option pretty much overrides all of these other Security Options and effectively gives the container access to all of the Privileges that it could want the readon flag specifies that even the container layer of the file system should be readon so if you don't need to have right access you might want to set this as another security Forin the security opt option allows you to specify either app armor or set comp profiles that you want to use for the container when it's run and then this last one is not one that you execute at container runtime but is something you would specify when you're running Docker D for example if you installing Docker engine and that is to enable namespace remapping for the user namespace allowing you to map from a non-root user on the host system to a root user inside of your containers as just one more layer of defense against a potential attacker so like I said these are very useful to know that they exist however if you're working on your development system it may not be as necessary to leverage more of these Advanced configuration options let's jump over to our editor and and configure Docker run commands and Docker compos configs to run our application and so I I have all of the necessary commands to do this stored within the make file here and I'm going to walk through how they're all configured and what they're what each of them is doing the very first thing that we're going to want to do is to build a version of our container image and tag it such that we can use it in our Docker run commands so this Docker build all make Target is calling a Docker build command and then because I'm located in a different subdirectory so I'm in the 08 I'm in the module 8 directory whereas we know our Docker files are contained in the in the module 6 directory and our source code is in the the module 5 directory that's why I'm having to use this file option and pass it a context outside of my current directory uh if you had your Docker compose and Docker files living alongside the code these commands would be somewhat simpler so this is going to go off and this first one is going to build my client react Docker file number three so if you remember as we were building up our Docker files at one point for the react client we were using using the npm Run Dev script and so that's actually using V to serve a development copy of our front end uh so we're going to build one image with that Docker file we're also going to use the final the final react client Docker file where we're copying a a built version of our frontend app into an engine X based container image so we'll have one tagged client react enginex based on that image we'll also build our node-based API using the final Docker file that we came up with and then we'll build our goang API as well using that final Docker file that we came up with for that one uh so let me just do make Docker build all and that will go off and build those four Images I did clear my image cache so this is going to take some time to build from scratch all of these images however in the meantime I can walk through the docker run commands that we're going to have to use to spin up these containers so I'm going to start by creating a Docker Network and so by default there's a number of Docker networks on the system uh let me just show those now so if I do Docker Network LS we can see these are all the default networks that are created we have a bridge Network we have a host Network and we have an empty null network if I don't specify a network it's going to get attached into this default Bridge Network however I want to create an isolated Network for my application and so for that I'm creating a network called my network so I'll do just that I can do Docker Network create my network and now if I list them again we can see my network and by default it uses that bridge driver which is what we want in this case the first container that we're going to run is the postgres container we've seen seen this command a number of times the only difference here is that I've added the-- Network option and passed it the name of my network and added the restart unless stopped command to tell Docker to restart it if it if it were to crash this- D flag will run this in the background we can see that postgres image is running in the background I could look at the logs if I wanted with doer log logs and then the ID we can see it's started up and it's ready to accept connections it looks like all of my uh four custom container images have now built so I can walk through and execute the various commands such that those containers will start up as well uh I'm going to start with uh API node so I'm going to work from the database forward so we'll have Docker run again we're going to run in the background with the DD flag I'm giving it a name I'm giving it a name here so that I can have another make Target that will clean up those resources afterwards I'm putting it on that same network so that it will be able to talk to my database I'm passing it this database URL and so I've specified that environment variable here and one important thing to note is that I gave the postgres container the name DB and then I've also added this D-Link DB option referencing the name of my postgres container and that gives me an alias with which I can address my database container and that is what is used here in my connection URL I should now be able to run Docker run for my node container we can see API node and postes uh at this point it looks like my node container is crashing for some reason so let's look at the logs here Docker logs so I wonder if this string is somehow getting escaped improperly so let me try again um Docker stop API node Docker remove API node ah it's because I copied and pasted this in without setting that environment variable export database URL equals that now if I run my container uh it should be able to pick up that environment variable and pass it in so I can do Docker PS uh it is no longer restarting and so I should be able to access on Port 3000 my API so let's make sure that is working and so it looks like my node API is working properly for the goang docker run command looks pretty much identical to the API node one uh except the name is updated and the image is updated we're still passing that link option so that we can reference the database in the same way and so let's start up our goang image it is now up let's switch over I know that one is running on 8080 okay goang is working properly and now we need to get our two front end containers so we're going to have the one running engine X and the one running uh V we'll start with the V one one thing to note is that my V config inside of Docker needs to be slightly different than my V config outside of Docker let me just compare the two so I'll select for compare and then here I will compare with selected and you can see when I was running these directly on my host without Docker I was able to my proxy was set up to just forward these to Local Host 880 now that I'm inside of that Docker Network I'm and I'm passing the the link option instead of referencing on Local Host I need to reference them by the Alias for that container so in this case it'll be API goang and API node and the way that I get this updated V config into my container at runtime is via a bind Mount so I'm using the- V option to specify that uh with my present working directory and then a relative path to client react V config.js and I'm mounting that to the same location that the other one would have already been located and this will overwrite the one that was in the container image with the one on my host system at that bind Mount location I'm linking this both to API node and API goang because I'm talking to both of them and then I've referenced my name as well as my image name down here in the command so I can run [Applause] this okay it's been up for one second uh so let's go to 5173 which is where I'm publishing that Port we can see it loading properly that's awesome and then the final container that I want to run uh is going to be my engine X based front end where I took my react app I built those files and then copy them into an engine X based image almost identical to the V one with the one difference being I don't need a bind Mount because the engine X config that I wrote was already designed to work within this containerized setup I'm also going to port forward from Local Host 80 to 8080 with ins inside the container which is where that unprivileged user from my engine X base image is is going to bind to by default I'll run that and now if I navigate to just Local Host 80 or by default if I don't specify a port it will default to 80 it is working properly and so with these with all these commands I can now spin up my application and spin down my application very easily I've added this docker stop make Target that will stop all of those named containers I've added this Docker remove one that will clean up those stopped containers off of my system and delete those as well as delete that Network and so I'll do that now and so now none of those containers are running anymore in my opinion having all these doer run commands with all these different Flags uh stored in a make file is pretty clunky and hard to deal with hard to modify hard to reason about luckily for us Docker compose exists and gives us a much nicer interface with which to do this exact same thing so I'll pull up this Docker compose file I'll also show you uh that composer eyes tool so let me just go to composer eyes zoom in a bit paste in my command click output and so we see my postgres command gets translated into this compose file where we have our services in this case it's named postgres we've named the container DB we're specifying the network we're passing an environment variables we're configuring the volume we're setting ports we're setting which ports to publish the restart option and the image I've gone ahead and done this for all of those commands that we had and that gener ated this Docker compose yaml file the latest version for Docker compose is 3.9 so if you're creating a new Docker compos file and want to take advantage of the latest and greatest features you may as well specify 3.9 and then within here I've got one block for each of my different services so I'll start with client react V and we can specify both an image as well as a build context and so Docker compos can handle not only running our our our containers but can also handle building them based on a context that's specified so in this case it's pointing to module 5 where the source code lives as well as a Docker file and here it's pointing to module 6 where the docker files live where the docker file lives this init true is the equivalent of a D- init in the docker run command again we've got our bind Mount here specified for the V config I did something slightly differently with the networking here I have two custom networks I have a front-end Network and a backend Network and so because my react app only needs to talk to my apis and should never need to talk directly to the database I've specified that those are attached to the front- end Network my react app and the two apis and then I have a backend Network which my two apis and the database are connected to and so that just gives me even more fine grain control over which Services can talk to which my enginex based service uh looks very similar the main difference being I'm pointing to a different Docker file and port forwarding to a different port my node API again quite similar I will call out I've added this dependson DB option and that tells Docker compose that wait until you've started up the DB container before you spin up the node container as I mentioned it's the API layer is connected to both the front end and the backend Network so we can have a listing of multiple networks here that this container should be attached to pretty much the same thing for the goang service and then finally the DB service which we're specifying that password it's only attached to the backend Network we've got our volume Mount at the known location where postgres stores its data and then we're using that public 15.1 Alpine image as our container image down here at the bottom we specify our different volumes in this case I have my PG data volume that's getting used for postgress and I specify my two networks because I haven't specified any additional data about these Networks they will use that default type of bridge which is the desired network type for this situation now with this all defined in this yaml file interacting with it is super easy let's say I want to build all those containers I can do Docker compose build and it will look at the different build sections for each service with the context and the docker file and go off and build those for me now if I want to run them I can just do Docker compose up and it's going to run all five of those containers and log out to the console from them all with color coding for each so I can see now I'm attached all my containers are running if I then go and load the page so in this case I opened up the engine X based front end off screen and we can see the logs coming from the engine X container followed by the log from the goang container where the front end made a request there followed by the log to the node container where the front end made a request and so we can see this be very valuable and help us to see what's going on within our containers by attaching to the logs in this way when we did our Docker run command we could have not used the detached option and instead had terminals connected to everything but with doer and compos it's just much easier to manage within a single terminal if I wanted to I could uh run these in the background with the- d flag I could do Docker compose Dash up DD uh and we can see it's created all those in the background if I then wanted to stop I would do a Docker compose stop and because my Docker compos file is named as such that's the default if I named it something else I would need to pass in a reference to that file uh with my Docker compose command just like when we run a Docker build command if we have a Docker file in the non-default location we need to pass that- F flag and so hopefully you can see the power of a Docker composed setup like this you'll be able to go to the GitHub repo clone it navigate to module 8 run a Docker compos up build and it will literally build all the necessary images and run it on your system and you'll have this application running in less than 5 minutes uh and you can see exactly how everything is configured within this one file and so this can be a super powerful tool for you and your team in setting up a common shared environment so that's it's easy to build and iterate on your your applications when we go to deploy the application and work on improving the developer experience we'll make some modifications to this and change the config a bit but I think this is a great starting point that hopefully highlights uh many of the different options that you'll use when you're running containers for your multi-service applications since many of those more advanced runtime options are security related it seemed like a good time to do a little bit of a deep dive on container security and this I think is my favorite of the memes that I created for the sections of this course just because something is running in a container doesn't inherently make it secure depending how you configure things at runtime it may be secure or it may not be so here if you specify user equals root the privileged option and connect it to the host Network you're removing almost all of the security boundary that a container would provide and it's about a secure is using that Cheeto as a lock there's two primary components to container security the first one is security of the image itself so as you're building the image what features are you using and how are you configuring things to make sure that your image is secure a big part of that are the software dependencies that are installed within your image and what attack surface area exists that a potential hacker could exploit the other key area to think about container security is at runtime so if someone were to successfully compromise your container because of some vulnerability that you built in what would they be able to do would they be able to escape the container and move laterally to compromise the host or other containers or would they mostly be stuck and confined within the container because you configure the runtime security appropriately some of my main things to think about when you're considering image security are how do you keep the attack surface area as small as possible one way to do that is to use minimal base images so the smaller your image the fewer things installed in it the less likelihood there is to be a potential bug that can be compromised one great source for secure base images is a company called chain guard which maintains a set of Base images specifically with a security Focus as you're writing your Docker file don't include things in the final image that you're not going to need if it's not needed at production time don't install it or use a multi-stage build if you need it at build time but not at production time also after you've built your image there's a number of tools available that you can use to scan that image for potential vulnerabilities there's one built into Docker called sneak there's another one from Aqua security called trivy take advantage of these tools to scan your images and see if there's potential vulnerabilities and what level those vulnerabilities are at this gives you visibility and allows you to choose if you need to patch them or if you're willing to accept having some known vulnerability in your image we saw this as we were building out our Docker files ideally you don't want to use a root user inside the container you want to use a Linux user with with the minimal set of permissions to accomplish the task at hand you should avoid building things like credal into the image itself you should treat your images as if they're public and inject any sensitive information at runtime you also can cryptographically sign your images to prove who was the person who built this and that can be useful to then verify on the other end that an image is what you expect and finally as you're choosing which base images to use make sure to pin at least to the minor version number if you Pinn to major. minor but not the patch version it would allow updates to the patch version let's say for bug fixes to be automatically rolled in but hopefully not incorporate braking changes into your system alternatively if you really want to lock down the base image you can use that image hash which will always refer to the same image on the runtime side of thing we can both think about the the container run time itself and the configuration options used for each container if we're setting up the docker demon Docker D within Docker engine one good tip is to use the user namespace remap option as described in the docs at that link to ensure that containers being run are in a separate username space than the host system and then for individual containers we can set the file system as readon if our application doesn't require WR access we can use the cap drop all option to remove all capabilities and then add back anything that we might need after that we can limit the CPU in memory to prevent a denial of service situation where one process is using too much of the resources and then finally we can use Security Options to set either set comp profiles or app armor profiles both of which you can find more information at in the docs throughout the course I've used a number of Docker CLI commands as we've needed them in this module I'm going to showcase the breadth of commands that exist and how you can use the help flag to get additional information the four main types of Docker objects that we'll be interacting with are images containers volume and networks all of which we've seen so far in the course for any Docker command we can do Docker d-el and it will give us information about all of the sub commands as well as a brief description of what those commands do in this case let's look at the commands associated with Docker images Docker images Docker image uh help and when you see all of the sub commands associated with Docker image we've seen the build command quite a bit it's how you take a Docker file and build an image from it the history command shows the steps that were used to build that image uh so I have a Docker file here that is just from Ubuntu 2204 run Echo hello command Echo goodbye and I built that into a image tagged with the tag New Image so let's do Docker image history on that new image and we can see my command is here at the top that was the last step in my Docker file I echoed hello here as the previous step and then these steps are the steps that were used to build that Ubuntu base image so for example I can do Docker image history on it and we'll see these steps within that history and so Docker history can be a good way to explore the steps that we used to build a container image the inspect command will show a whole bunch of detailed information about images it can actually be quite overwhelming uh so we can do inspect on that same image and we get this big Json dump with a whole bunch of metadata associated with that container image the import command allows us to take a tarball and create an image from it honestly I've never used that before the load command similarly can take a tar archive that was generated using Docker save on a container and create an image from it I don't find that very useful all my images I'm generally building from a Docker file image LS will list all the images on our system and so as you can see I have a whole bunch here some with tags some without tags uh prune is a great way to clean up those old images if I wanted to clean up my images I would do doer image prune it would remove any image not associated with a container that's what it means by dangling images I'll skip that for now the pull command we've seen is how we can pull an image from a registry we haven't used it directly that much because when we do a Docker run if the image doesn't exist it will first execute a pull but if I did want to execute that I could do Docker image pull and then an image name and tag and it would go off and pull that from the registry push likewise pushes our our built images to a registry we've seen that one quite a bit so far if I want to delete a single image rather than using prune to delete them all I can delete just one with Docker image remove new image for example now it's untagged and deleted from my system the save command can take an image and dump it out to a tar archive this could be useful if you wanted to have if you want to be able to explore the file system uh outside of a container I haven't found this to be that useful in my workflows and then finally we can use the docker image tag command to tag an image with a new tag so I could do Docker image tag then to 22.4 mytu tag Docker image list GP mytu and we can see I now have an image which is based on that Ubuntu 20204 but it has my custom tag associated with we saw this when we built the image with one tag and then tag them with new tags to push them to those to Registries in the container registry module that's it for the sub commands of Docker images however there's one more command that's associated with images that's not a sub command and that's Docker scan so I can do Docker scan uh let's just scan the aun2 image for fun 22.4 and that will use a tool called Snick to go off query of vulnerabilities database and assess whether there's any known vulnerabilities within that image it then outputs a report of all the different vulnerabilities that we've found things like system D op SSL uh these are at the medium severity level we've got one here at the high severity level it also tells you when in the Upstream project that vulnerability was fixed so then we can decide do we want to use this base image or do we want to pick a different one that potentially has this mitigated already that covers most of the things that you'll be doing when interacting with images let's look at the options for the docker container command now as you can see there are many more sub commands for Docker containers than there were for images I'm just going to highlight a few that I think are useful to know uh the attach command if you have a running container already and you want to attach your local shell to the input output and error streams of a container you can do that with attach a similar one to that is the exec command where rather than attaching to an existing stream you're going to run a new command within the container just like with images we can run the inspect which will dump a Big Blob of Json metadata that can have useful information the Stop and kill commands are how we can either gracefully or non-g gracefully uh stop a container so if you have a container that uh you've tried to stop but it won't die you can always use kill and that will generally do the trick you would want to issue that with either the name of the container or the ID of the container so if we do Docker run uh Ubuntu sleep 999 now that's going to sleep for a th seconds and even if I do control C it nothing's going to happen uh so if I want to get out of that I can do Docker PS uh we see that container here is running and then I would just generally do Docker kill paste that in and now our containers exited and we've got our shell back if you want to view the logs of a container you would do Docker container logs and then either the ID or the name of the container if you want to Trail the logs so if you want to have a live feed of them you can add the- F command this container here that you you've seen pop up when I do Docker PS it's a system container that you'll likely see on on your host but let's just check the logs of it since we're here we can do Docker logs of that container we also can do Docker container logs those are the same command and then finally if we wanted to tail those logs we can add the- F and now any additional logs would be printed to the console automatically the ls command lists all running containers and I believe is pretty much identical to Docker PS if we add A- a flag it will show stopped containers as well so without the d a it's only running containers with d a it shows stopped as well if we wanted to clean up all those stopped containers we can issue the prune command or if we wanted to remove a single container we can use the RM command the Run command is one we've seen over and over where we're creating a container from a container image top can also be a useful one if we want to see what's running inside the container so let's do a Docker run- DTU sleep 99 and now we have this container idid docker Docker top Docker container top then ID uh and you can see the processes running inside of that container and then finally the weight option would could be useful if you're writing a script where you needed the one container to finish before you progressed you could issue a Docker container weight command which would stop your script until that container exited then print out the exit code and you can proceed on with your script from there the volumes options are much simp we only have these five options we've seen the create option where you just pass it a name Docker volume create my volume we can use the inspect volume to look at that we can see where it's mounted within that virtual machine file system the name the driver Etc the ls command will list all available volumes marker volume LS we see a few named volumes here at the bottom we also see a bunch of anonymous volumes above it that are created and managed by Docker if you have data that you want to persist it's much easier to manage if you go with a named volume create that volume and then mount it into your container at runtime just like with images and containers we can use the prune command to remove all of our unused volumes or the RM command to remove one or more volumes now finally let's look at the options for networks they'll look very familiar given that they're almost identical to the volume options Docker Network just like with volumes we can create inspect list prune and remove the two new ones here are connect and disconnect and so if we had previously created a container but we forgot to connect it to the network that we wanted with the Das Das Network flag we could then run a Docker Network connect with the ID of the container to tie it into that Network so we can do Docker Network LS um let's say for example we wanted to do Docker run vuntu sleep 99 I'll run that in the background but we didn't want it to be on the bridge default network instead we wanted to connect it to my network so that it'll be able to talk to other services that were spawning there we can do Docker Network connect my network pass it the ID and now if we inspect my network we can see the listing of containers that are active there and we have this one named awesome Brown just confirm awesome Brown is the automatically generated name for that sleep container that I created if I wanted to remove the container from that Network for whatever reason I could use the do Docker Network disconnect command uh generally with these networks because I'm bringing containers up and down so frequently rather than starting a container and then attaching it I would just want to add that to my Docker compos config or as a option in my Docker run command so that it would be connected to the proper Network at runtime now that we're fully versed in the docker CLI how we can make the developer experience working with containers extremely nice what are the main considerations that we should think about in terms of the developer experience when building out applications like this how do we go from this confused frustrated developer at the top this excited pair of programmers here at the bottom we want an easy and simple setup we want to be able to iterate without needing to rebuild the container image so far even if we made the tiniest change to our code we would need to rebuild the container image and then rerun the container to have that reflected and that's just a terrible experience that being said we can can use the bind mounts that we learned about to mount our source code from our host system into the Container at runtime and then we can use hot reloading utilities like nodon or there's one for goang called a such that with each change of our source our applications will get automatically rebuilt and restarted and those changes will be reflected immediately we need to be able to debug our code as if it was running locally and for that we can add debugging utilities into our development image and set up a special Docker compose file to use those utilities we also want to be able to execute our tests within the container again I can make a custom Docker compose file that we'll use to do that in most scenarios the only containers that you'll build be building on your local system are the ones you're currently developing on for production images you'll want those built within a continuous integration Pipeline and finally it can be super powerful to be able to spin up an ephemeral environment with each pull request that you make that will have all of your modifications built into it automatically that you can share with a team member to get feedback and iterate on things more quickly when we get to that that's where our sponsor of today's video Shipyard dobu comes into play they make it very easy to take your Docker compos setup and get a fresh ephemeral environment built and deployed automatically with a simple GitHub event like a poll request so let's jump over to the code editor and take our initial Docker compose file that we built a few modules ago and modify it to improve the developer experience in these ways so we first built out our do ER compos file in module 8 so I'm going to copy that into module 11 let's just make sure it still runs I already have a make Target set up that we'll call uh compose up and I'm going to name this Docker compose Dev make compose up build which will look at that Dev compose file build my images and then run them uh it looks like uh line nine here where I'm using my present work directory it's no longer a valid reference to that V config file in instead I'm going to have to set the relative path to that like so okay so things appear to be working as expected just had to make that one tweak to the volume Mount i' would say if we're if we're thinking of that wish list that I just had up easy and simple setup is a check we're able to get up and running with just that single command now we want to improve upon iteration without having to rebuild the container also while this volume Mount syntax works I slightly prefer the more verbose version that we can specify here with a type A source and a Target so this should do exactly the same thing and now in addition to bind mounting in that configuration let's also bind Mount the entire source s code directory so that's going to be uh in this module 5 client react and so this will bind Mount the client react directory into our container at user Source app one additional point though is that within client react we have things like the node modules directory and the uh distribution directory which we don't want to get bind mounted in and so the way that we can overcome that is to specify another volume Mount that has no source and only a Target and so by doing this it will effectively overwrite this bind mount for this particular path and avoid bringing those node modules from my host system into the container and just to remind myself what the command is for Docker file 3 let's go look at that one client react Docker file 3 this is running npm runev which in turn calls V so we should have hot reloading built in and so now with these bind mounts in place let me start up my application again now if I navigate into my source code and make a change let's see what happens client react source app.jsx save it because we're bind mounting that Source in we automatically reload our application with feet and that gets built in and we see it immediately on the front end now let's take the same technique and apply it to our two API servers here we're going to use the same bind Mount approach for the uh node API including the empty volume Mount over node modules to avoid bringing those into the container and I'll put it right after in it true uh instead of client react we want API node and let's just remind ourselves what this Docker file 8 is doing okay so we're going to want to make a few tweaks here uh this Docker file is optimized for our production deployment but in this case we want to have a separate build stage that we can Target for this development environment I'll start by naming my first stage uh as as as space and then where we diverge from our production versus development is going to be at this step where for production we only want to install the production dependencies so I'm going to say from base here and then I'm also going to have uh as as as production and then I'll do from base as Dev and so now within this stage we can have our development commands so our development commands are going to look similar uh but instead of npm C only production uh we'll do npm install and that will install both Dev and production dependencies let's also move this environment variable down such that we set that only in the production image after we've installed our dependencies we want to copy all of our source code in and then for the development stage we want our Command to be uh npm runev uh and I'm actually going to call this Docker file number n just so we can keep them keep track of them uh so we have our Dev stage where we're installing development and production dependencies we're copying our source we're running npm runev and then we have our production stage where we are doing what we were doing before with the non-root user installing only production dependencies uh Etc now in our Docker compos Dev let's use that new Docker file 9 and see if we succeeded oh one thing I forgot to do is in addition to a context and a Docker file you can pass a Target and that's allows you to specify which of your stages in your multi-stage build you want to Target so in this case it would have defaulted to production as the final stage I want it to Target that Dev stage because that's going to be where we're running nodemon for the auto restart so now let's make a change to our node API Source here Source index uh in our response let's just add add add Tada so we save it now refresh and we see our node reserver was detected noemon saw the change restarted uh and now when we hit that API from our front end we get the updated version back and so we've got hot reloading working for our front end in vit as well as our uh node based API let's do the same for goang our bind Mount setup should be quite similar a volume going to API goang uh we don't need the node modules Mount depending how you have your go path set up uh that will determine where you're going to install your dependencies on your host system uh in this case it's outside of my project directory so I don't have to worry about shadowing that with an empty volume however my target inside the container instead of user Source app if we think about the work directory that I used in the docker file it's just slapp uh and now I need to do something similar to my production Docker file add some development stages so that I can have both my production ready and my development images make a new one called Docker file number8 make sure we're using the new one and we're going to add a target of Dev and we will get around to adding that here uh so my first stage already has a name but now because I'm going to have I'm going to split this into two build stages so I'll have build base as this it should be the same up until here where we download our dependencies however this is where we want to split and so I'll do from build base as build Dev and so for our Dev build in addition to building in addition to pulling in our dependencies that we've specified there's two more dependencies that that I want to add from a development perspective and that is a package called air which enables a hot reloading so it'll detect a change like noon rebuild that application and then restart it as well as a package called delve which allows for remote debugging that we'll set up so we can uh do set break points and connect to our container remotely to investigate the state of all of our variables and that sort of thing and so the command to install those two dependencies is go install pass it the name of the package and go install pass it the name of the package we still need to copy in our source code and then we'll set a command using that air package so we'll call Air the and pass it this default air.

toml which I created ahead of time and it just tells it sort of thing things like where the main file is what command it should run when it needs to rebuild where to look for changes Etc and so that can be our build Dev stage uh that will be actually I'll just call it our Dev stage like we did in the docker file and then for the remainder uh we build our production images and then copy them into that final scratch image so I'll do from uh build Bas as build production then down here when we copy from build instead we're actually copying from build production uh we'll add this non-root user within the build production stage and so that should be good we're now targeting our Dev stage we're bind mounting in our source code we're using that air package to do automatic reloading uh let's try it out okay it looks like our app has started up so let's uh load here uh we can see it's running let's make a change to our goang API just to see that rebuild happening if we go up to module 5 main.go and add something here we see it detected main.go is changing uh it just rebuilt and so if I reload we get goang woo awesome I'll save it again you can see it's detected it it builds it I refresh that update is reflected so now we've got hot reloading within our containers by bind mounting our source code in and then using utilities like nodon vit or air to automatically detect file system changes and update our application in real time now the next item on our wish list is remote debug so we want to be able to run a debugger within our containers and then connect to it from our development environment as much as we love doing print F debugging where we put in a bunch of print statements sometimes you really need to use a debugger uh and investigate a little more thoroughly so let's get that set up now what I'm going to do is create a new Docker compose file that I'll call Docker compose debug and one interesting feature of Docker compose is that you can take two files and pass them both into the docker compose up command and it will take the first file and then take the second file and overlay it you only have to define the fields that have changed between the two you can have your base configuration and then slot in slight differences and so in this case we're going to be able to leverage that Docker composed Dev file almost entirely but we'll add a few new Fields mostly adding additional ports to publish because we're going to have to connect on New Ports that the debugger is listening on and modifying the command that's executed when the container started uh we'll start with our our our version of 3.9 3.9 3.9 Services uh the services that we're modifying here are the API layer so we'll have API node and API goang we want to override the command field so we'll have a command here and so instead of npm run Dev like we were running before uh we actually want npm run run run debug Docker and I added that as a new npm script so we can go take a look at what that is npm debug Docker runs nodemon so we still get that hot reloading uh but the Das Dash inspect command tells node to run its debugger here we're listening on all hosts on Port 9229 so this will enable us to connect from our Local Host system on Port 9229 as long as we open as long as we publish 9229 in our composed file and then we're passing it that index.js file as the entry point for the program uh so here we'll do ports and I'm going to have both my normal 3000 Port that I had before as well as my new debug port and so when I run this command if I do a Docker compose debug build I do Docker compose and then I first pass it my development compose file so that's my Docker compos ddev and then I pass it my debug compose file Docker compos debug and Docker compos takes those two and interleaves them together such that these will be the new values but we can still take advantage of all this additional config here our goang API will be somewhat similar the debugger that we're running for goang uh runs on Port 4000 it's a tool called delve which was that other Dev dependency that I installed and our Command I'll just copy and paste it here because it's a little bit long uh uses that utility in debug mode we pass it the main.go file that should run we tell it which Port we want to listen on as well as some additional configuration options and so that should be enough to run our application in debug mode so I can do make compose up debug build as I showed it passes the dev file first then the debug uh composed file so while that's building and setting up I can show you the launch. Json configuration that I use and that is the way that VSS code sets up its debugging utilities so under the configurations block I have two of them I have one set up for node with Port 229 one set up for goang on that Port 4000 uh the key things to call out here are that I'm using the request attach mode because my container will already have that debugger running inside I don't need to spawn a new one I just want to attach to it and then this local root command tells VSS code where the root of this project is within my workspace and that's so that when I create a breakpoint it's able to map that to a path within the container which I'm passing here of user Source app oh I see I hit an error here it's because I forgot to change uh this port for the goang API to uh 8080 so let's try that again similarly for goang uh we're using a particular adapter that is associated with that delve again we're using the attach request option specifying the port giving it a a remote path and then this one's this one was a little more tricky to get set up again we need to figure out we need need to be able to map from our host system into the Container such that it's able to know when we set a break point what that corresponds to and properly apply it inside the container and so the substitute path command takes the path on the local system and modifies it such that it gets fed in properly inside the container my app is now running in debug mode I don't have any breakpoint set or anything so it just loads as normal however I can go into the run and debug portion of my uh code editor and pick one of these so let's pick attached to node click the play button aha uh it was not connecting because it was forwarding my debug request to Port 3000 where the app is listening and so instead I should be forwarding to the the the port where the debugger is listening now that the API have started up I'll click play and now it did successfully attach we have our breakpoint in in our source code just before it sends that response back if I reload here we see it caught at that break point and we can see all of the local variables including the response itself Wednesday February 23rd API node 1257 so if I then click skip over the breakpoint or play it will resume and allow that to be returned from the API so let's click play and we see we got back that response I'll click disconnect here and let's try the same thing with goang click the play button looks like we were able to attach let's load our goang API and set a breakpoint somewhere within our goang code under main.go uh let's do right before we send back our data breakpoint we can see it being injected here in the logs and now if we load the page once again we successfully caught at the breakpoint uh and we can see the locals here uh we have our time data as well as our gin context and now we have debugging working for both uh the goang and the node-based API uh I also added a react query debugging tools as a de as a Dev dependency that get built in here allows us to get all sorts of information about the uh queries that react query is making to the back end uh so for example if we load here we can look at this query when it was last updated uh the data that we got back we can tell it to refresh refresh the data and so without having to reload the whole page we can interact with the react query specific components directly that gives us debug capabilities for all of the services that we're building so that's awesome and now the next thing our wish list was to be able to execute tests with inside the containerized environment similar to how we had a overlay compos file for uh debugging I'm going to do I'm going to make us another one called Docker compos test uh we don't need the port access we will be overwriting the command uh we'll just execute our npm run test script which will execute just which in turn looks for our test so I can just show you the test Suite that I have set up it's it's quite minimal so API node test we have one example test that expects true to be true so that's always going to pass and the package.json specifies uh test calls just which will discover the test and execute it and then for goang similarly minimal uh I have a test module with exampl test. go uh it tests that 1 equals 1 which indeed it it it does and my command that I'm going to use there is go test so here instead of using delve I'm going to run go test and then I'll pass it this V option which will enable it to properly discover those tests so now in order to run this I have two make targets set up or I have uh Docker composed calls for goang and node separately so again I'm going to use the dev compos file first overlay that test compos file onto it and then I can specify run build and a specific service within that compose file and so this one should execute the goang test and this one should execute the node tests so if I do make run tests it's going to build those images and execute those tests so we see the just test passed and let's see let's go find the yeah here's the goang test here so the goang test pass the just test passed and we were able to execute those tests within the containerized environment the next item on our wish list was continuous integration uh and so that essentially refers to this idea you whenever you're pushing code to your virsion control system you should be running pipelines to do things like execute your test Suite uh build and push your images Etc GitHub actions is a continuous integration system that's built into GitHub and you can just add a couple of files to your repo and it makes it very easy to set up this type of thing I've added a workflow to the repo that I can show here that is going to do a few things it's going to build our image it's going to scan it tag it and push it to dockerhub um I'm only doing this for one of the services just as an example but you can use the same Concepts to add it to as many services as you want within the repo you place GitHub action files at the GitHub workflows directory and then any yaml file in here will be interpreted as a workflow with about 50 lines of code here we're able to configure a GitHub action that will build our images scan them test them Etc So within this file we have a number of different sections I'll just walk through them now this first section starting with on tells GitHub when it should run this workflow in this case I've specified you want to run it on push events to specific branches and tags so if I push to the GitHub action branch which I'm on now uh it will trigger this workflow or if I push a get tag with starting with the letter V so maybe it's vx. y.z so some sver version when I do a release you can also do other types of events like a cron string so you would run it periodically based on a schedule or upon creation of pull requests below this you define your job section where you have the different jobs that you're going to run and then within each job you can have a number of steps and so I've only defined a single job I'm just going to build tag and push my image you specify which uh runner you want to run on I'm running on auntu latest and then the steps that I'm going to run are executing first checkout which is a public action provided by GitHub which checks out the code I then am using a public action from Docker to generate a bunch of tags that I want to use for my image using information such as the name of the branch the name of the pr the version if I'm pushing a tag I can use that to generate a Docker image tag and then finally I'm using the date and the commit hash as another example of a potential tag that I might want to use like I talked about when we were looking at container Registries here I'm giving it the name of the repo that I'm going to push to and these actions these public actions that I'm referring to are available in the GitHub action Marketplace and so this was an action that was produced by Docker um you can see the source code here for it but if we go to the action Marketplace uh we can see here it is it tells us all the different ways we can use it the types of inputs the types of outputs Etc so I'm able to specify that this step should use that action give it these types of tags and will automatically generate those for me next again I'm going to use a public action from Docker to log in dockerhub and within my repo I can specify my username and my access token such as it's table to do that so within the repo within the repo under settings under secrets and variables actions you can specify environment Secrets or repository secrets and so here I specified my dockerhub username and a access token such that that action is able to log in next up again another public action from Docker so really I'm I'm leveraging their work to make my life super easy this one is going to point to the file that I want to build as well as the context with which I want to build it I'm saying I do want to push it to dockerhub and then I'm using the output of my metadata step here and applying those as tags so this is going to be a listing of all those tags that are generated this will apply them all and push them all and then finally I'm leveraging a tool from Aqua security using their vulnerability scanner to look in the image for any critical vulnerabilities you can specify what level you want to block on here and so I'm saying if there's any critical vulnerabilities in the image fail the run and that way I can go investigate uh if there are no critical vulnerabilities it will pass the run so I haven't made any changes on this Branch uh but I do want to trigger a run so I'm going to do an commit I do get commit DM empty allow empty get push and now if I go to my repo under actions we can see that empty commit just triggered a build here's my job if I click into it I see it starting up that Runner and and then each of my steps we can see it's checked out my code it's generated the tags so let's look at the tags that it generated we've got GitHub action is the branch I'm on uh latest is just the default and then this is the this is today's date as well as the short hash from the commit we successfully logged into Docker Hub we then successfully built and pushed our image we ran the scanner and found no critical V vulnerabilities and then there's just some cleanup at the end so now let me go to dockerhub we should be able to see for our API node repo this version was just posted a few seconds ago and so that kind of highlights The Continuous integration workflow that you would want to set up for your container images that's just a taste of what you can do with GitHub actions and with continuous integration pipelines uh Brett fer has a repo called Docker cicd automation that has a whole bunch of really cool examples around uh doing things not only building and testing and pushing your images but actually interacting with PRS adding comments to those PRS with the results of the tests and that sort of thing you can see his talk that he gave on this on YouTube I would definitely suggest checking that out now the final topic that I want to cover here in the developer experience section of the course is ephemeral environments ephemeral environments are this idea of building buing and deploying short-lived isolated environments that we can use for any sort of purpose including testing validation QA Etc and in particular I'm going to use this platform from Shipyard to very easily set up this for our sample application I'll start by just walking through their onboarding process and I'll show you with just a few lines of configuration to our Docker compos file we can have ephemeral environment set up with every new PR to the repo I'm here on the shipyard dobu website I click log in It'll ask me to log in with my gift Hub I'll do that you'll authorize Shipyard to access your account this now redirects me back to the shipyard site where I will connect my GitHub organization so that initial connection just gave them access to my user information now this is the authorization to actually access my repos you can choose either all repos or only select repos so here I'm going to pick the specific repo that we care about which is the docker course repo this will grant them the necessary permissions to the contents of the repo they also have features where they can provide information back on the GitHub interface that's why it has read and write access to checks commit statuses and pull request at this point I'll just click install now as we can see we're on the free trial period as I mentioned during the introduction students of this course have a exclusive promotion code where the first 300 people to sign up can get an extended trial period so go check out the description for more information on that and then at this point we can click the we can click this scen button to add our first application we'll select the repository that we care about it's our Docker course repo and for our base environment I'm going to use the main branch we can use the default name for application and now if we click the select Services button it's going to scan the repo for a Docker compos file in this case because of the structure of the repo based on the course it wasn't able to find specifically which one we want I'm going to choose the one that we set up initially in module 8 because that's the simplest Docker compos file that contains all the services we care about I'll just click select and now we need to pick which of the services in that Docker compose file we want to include here I'm not going to include our V based deployment instead I'm only going to have the engine X based front end um we'll leave on our two apis in the database here and then we can see that it automatically Det Ed our named volumes and an interesting feature here is that because we have that named volume whatever's stored in that volume on our main environment will get snapshotted and copied over to uh additional environments as we make PRS against that at the top here it's saying please include at least one service with a route and so we need to provide Shipyard a bit more context and configuration so that it knows how to route traffic to our various services and the way that we do that is by adding labels to our Docker compos file so let me jump over to the code code editor and add in just a few lines to our Docker compost file so that Shipyard will be able to properly detect and utilize those routes and so each of our services we can add a labels field and so here under the engine X client react enginex field I'll add a labels Target and add these two labels the first one is specifying that this is the primary route this is the route that we want traffic on the root path to come into uh and then we're specifying the route as just the forward slash so this is saying this is going to be the default service that we want you to Route traffic to and so with those two things that's good um we'll adapt for for for our API node Service uh we'll add this this is not the primary route so we'll get rid of that and we will add this going to slash API slash uh node just like we had before and then similarly for our API goang we'll go here uh and just replace node with goang and so with just those I think it was additional seven or eight lines of uh code into our Docker compose file we can now commit that to the main branch push it now if we go back to Shipyard and reload our latest commit now we can see that it automatically picked up those labels and it's including those in the information about our configuration here with that I'll add environment variables all the default values here should be fine we don't need any additional environment variables and click create application this is now automatically going off and building all the containers based on the latest commit to that main branch it will then bundle them up and deploy them now you'll notice that the first time you log into Shipyard you'll get this message at the top that's because it's provisioning a standalone kubernetes environment for each customer uh and that allows for isolation between customer data however it does take a few minutes to set up and so the very first time that you you're here you'll have to wait about 10 minutes after that the build and deploy should be much quicker now while we were waiting realized that I should have added one additional configuration here uh for our API layer and so that is that is the shipyard route rewrite label and this is saying if we hit the URL API node our node service is expecting that to come in at the root path and so by adding this label it will strip out that prefix uh and we'll do the same thing for goang because the routes that I've defined inside of my services are not expecting that prefix so I'll add those and push them now another neat feature that we can turn on is if we go here to this uh three dot menu and click under configure under notifications and general settings we can add GitHub commit checks and GitHub PR comments so the when Shipyard has information for us it will populate that back into GitHub so that we can see it from there we also can tell it which pull request we want to deploy on so we could have specific labels let's say we only want to deploy on bugs or we only want to deploy on enhancements or we can specify a r a regular expression to decide which branch names we want to deploy with in this case I'm going to leave these as the default so that should deploy any PR back to the main branch I'll click save there's also a slack integration you can use uh to get updates and notifications from chipyard I'm not going to go into that now but I just wanted to call it out as an option a few minutes later back on our home application page we can see that our base environment has built and is ready to be visited now you'll also see this little timer here that says stopping in 2 hours and 49 minutes so that's a feature that is called SLV or since last visit and you can configure that within your organization to tell Shipyard how long an Emeral environment should live and by putting environments to sleep if they're not being used allows you to have more ephemeral environments on a given tier within their platform if I click the visit button it's going to take us to this page where we get a login prompt so we can log in either using GitHub or Google for ooth and this allows you to have these ephemeral environments not public on the internet but actually protected behind this login layer where you can very finely control who has access to it so I'll click to log in with GitHub and we can see we've got our application running within their environment and if you look up here at the URL we see our custom URL that's associated with this particular ephemeral environment now if we wanted to if we wanted to Grant access to this we could go to the visitors Tab and under here I can add GitHub usernames or Google usernames and that way you can control who has access via single sign on so that's our base environment now let's make a slight change to our application and then redeploy base on a pull request event I'll go back to my code editor uh and let's have it deploy with a different message on that home screen so I'll go into my example web application here on the front end in my app.jsx file I'll scroll down and instead of hey team we'll say hey team from Shipyard and we'll change the wave into a ship awesome uh now I'll do get checkout DB Shipyard demo and I'll commit to this new Shipyard demo Branch change message now I'll push that now if I go over to GitHub I can see that recent Branch had a push I'm going to create a pull request from that Branch back to the main branch if I click create pull request and now jump back over to Shipyard we can see it immediately detected that pull request and is now building a new ephemeral environment specifically for that new Shipyard demo Branch once it finishes building we'll be able to see that is different than the original Emeral environment with that updated message clearly a change that small is quite trivial but it could represent any number of complex changes to our different Services we could be changing the API layer we could be changing our things like our database schema let me go here to the build details dashboard and show you how that's progressing it shows us the various services that we have within our application that we selected initially we can see the build logs as it ran our Docker build commands and looked within the docker compos file found the corresponding Docker files and built those out we can see all the logs here across our different Services now under the deploy tab we can see the pods as they come up and then under the Run tab we can see the logs coming out of the containers themselves it's now handling that final DNS setup so that we'll be able to access the environment and now we have our second ephemeral environment live let me visit that one as well as we can see it's has taken that change that we made in that PR and deployed it in a separate environment from our base environment there's an additional interface here if we click terminal it'll pull up a way for us to get additional information about all of our environments so we've got our main environment we've got our Shipyard demo environment let me just click into the main one and we can see the pods running here within that environment I can choose one of them if I select it we'll get the logs here at the bottom we can also hit the r key if we wanted to exec in and do some debugging that way and so now this is a shell running inside that pod within the environment I'll exit out there's also a CLI that I can run locally to get a lot of this functionality on my host system directly from my laptop let me jump over and show you that CLI real quick to install the CLI uh they provide either a bash script that you can use or you can use something like Homebrew to install it I've already got it installed and then the next piece is to get your Shipyard API token and so in order to turn on the CLI for your account you'll need to ask the team at Shipyard to do so but once you do you can go up here to the user and then click profile and it will show up on that page and you can grab that API key to use with the CLI we can see here all the different functionality that we have so I can do something like Shipyard get environments that's going to list out all the environments that I have and let's say I want to get the logs from one of my pods like I could see from the web interface I can do Shipyard uh Logs with the N flag I can pass The UU ID of that environment and then with the service flag I can tell it which service I want so let's do API uh node and we can see the log's coming out of my node based API application uh similarly we could port forward directly to one of those applications so let me just port forward to that one so instead of logs I'll do port forward uh and then my node application is listing on Port 3000 so I'll do 3,000 uh it wants the port flag and now my Local Host 3000 should be forwarded to that node-based API and I can go here and I'm connected directly to it this would be super useful if I wanted to do something something like uh maybe connect to my postgres database directly with PG admin or otherwise I could port forward to that environment uh make whatever changes I needed to let's say I was building out my base environment and I needed particular seed data and I wanted to run my migrations I could do that manually once on my base environment and then those changes would get snapshotted because they're in a named volume and built into my additional environments moving forward so hopefully this gives you an idea of the power of ephemeral environments and the ease with which we were able to use the shipyard platform to generate them I created That Base environment from my main branch and now every PR that I create moving forward we get an ephemeral environment to validate before we deploy to staging or production if you a call we turned on the ability for Shipyard to populate information back on GitHub so let's go see what it did on that PR if I go under pull requests demo Shipyard we can see that the shipyard app created this comment and tells us the environment was successfully built and running we can access it at at this link or see the build details at that link and so this just makes it very easy for someone reviewing this poll request to now go check out what exactly these changes are and how they impacted the application hopefully that demo kind of give you an idea of the power of ephemeral environments and why we would want to use them within our software life cycle and showcase the ease with which we were able to use the shipyard platform to build those out for our project at this point we've got all the tools within our system to have a really nice developer experience with containers and now now let's shift gears and actually learn how to deploy them into the cloud all right congratulations we've made it to the final module of the course and that will be focused on deploying containers so we now know how to build container images run containers from those images we've learned about how to do development with them and now we need to build on those skills and learn how to deploy our containers to production now one of the main benefits of containers is that they provide the standard interface upon which to run as long as you can provide a container image that meets that open container spec there are many cloud service providers that will happily take that container image and run it for you I saw this article back in 2021 from Corey Quinn who runs the last week in AWS newsletter talking about the 17 different ways to run containers on AWS now obviously AWS is kind of this sprawling massive cloud provider but even still to have 17 different ways to run containers uh ranging from container-based serverless options like Lambda uh to elastic container service to running kubernetes with elastic kubernetes engine uh there's just so many different ways to run containers and this is just within one cloud provider and so that portability is both a blessing and a curse meaning that it provides us this flexibility but also it can be a bit overwhelming in terms of deciding which of these methods to use and now what do we actually care about when it comes to choosing a method to deploy our container based workloads I'd say the first item that's top of mind for me is security and whether either the cloud provider is handling security out of the box and setting all those container runtime options such that it meets the threshold we need or they give access to those knobs and dials that we can set them ourselves the next one is the ergonomics or the developer experience so how easy is it to go from uh having a set of container images and running those and getting traffic from the internet to them also how easy is it to deploy new versions so if I make a change to my code how does that code make its way into production and then the third one in terms of ergonomics is how do I work with the running containers how do I check the logs uh can I monitor them can I collect traces easily combining those three elements make for either a good or bad developer experience when when working with any of these platforms the next one is scalability so does the platform meet my needs in terms of the scale of any individual container or the number of total containers required for example can I access particular types of gpus uh what if I need to scale beyond the capability of a single host can I launch containers across multiple hosts and how do those Services then communicate with one one one another another key consideration is persistent storage so we've talked about vol volumes within Docker we've mostly focused uh from a developer perspective with Docker desktop and seeing how those volumes get stored on your laptop or desktop uh however if we're now running containers across multiple hosts and they're getting scheduled and they could be scheduled on one versus the other we need to think about how storage works and if it's possible for those storage to be accessed from different virtual machines or different physical machines and the last one that I have here is cost how affordable is it to run your containers in this particular environment these range from sort of bare metal where you're paying just for the compute but handling all the operations yourself all the way up to maybe a fully managed platform as a service offering where they're charging you a premium for some of those improved uh ergonomics or developer experience features so thinking about how much it's going to cost you understanding the financial impact of your platform will be important if you're running a business based on these containerized workloads now when I was planning out the course uh I actually built out three different deployment targets so one using railway.

apppp one using Dockers swarm and one using kubernetes uh since we're already at about 4 hours into the course I've decided to shift that slightly uh within this video I'm going to demonstrate the docker swarm deployment and then I'm going to create separate bonus videos uh showcasing how to deploy onto Railway and how to deploy onto kubernetes for the the sample application that we've been working with uh and once those bonus videos are released I'll have information in the description about how to access them now at this point you might be asking why can't we just take the docker compos file that we created in module 8 uh create a virtual machine and run that directly now while that would work and you would have your containers running on that host and you could set up the firewall to access them it's not ideal there's some major limitations with Docker compose uh that prevent us from wanting to use that for production workload for first there's no way to deploy a new version of our containers without having to stop and then restart the container and so by definition you're going to have some downtime there it might be very quick on the order of a second or two uh but nonetheless you generally want to avoid downtime if you can and then also there's no utilities for easily rolling back a version uh with Docker compose directly also Docker compose uh doesn't have a native way to handle credentials properly we've been passing in the that database URL or the postgress password as an environment variable there's no there's no mechanism in Docker compose to handle encrypted secrets and so that's a major limitation there also you can only run Docker compose on a single host uh so that hasn't been an issue as we've been developing on our development machines however we would like the ability to expand Beyond a single host and add things like high availability in the future and so Docker compose is limited in that way there's two other reasons you might have historically heard cited for not using Docker compose in production uh the first is that it used to not support health checks and so we'll talk about what health checks are and add some in that is no longer true they've added health checks to Docker compose relatively recently and then the other one has also been mitigated is that Docker compose used to live as a separate binary uh when you would run it and so it was just harder to install and config now that Docker space compos rather than Docker Dash compose is built into the docker binary itself that's no longer an issue either you may hear people reference these as potential reasons just know that historically they were reasons not to use Docker compose in production now I've phrased these as things that Docker compose doesn't have however as we'll see these are features that Docker swarm does have built in we can configure a deployment strategy to have zero downtime deployments we can roll back changes if we want to we can handle credentials via a feature called Secrets uh and we can scale a swarm cluster across many hosts to keep things simple I'm going to deploy a swarm cluster on a single host uh so it's going to be quite similar to the conceptually to using Docker compose up uh we're going to create a swarm cluster consisting of a single virtual machine and then we're going to modify our Docker compose file and you'll see that the actual changes in the configuration are quite minimal uh there's only a few things that we need to tweak to make our composed file compatible with swarm uh we're going to deploy that and then we'll set up DNS so that we can access it via the Internet what's the process going to look like I'm going to create a virtual machine I'm going to make sure that the firewall set up for that virtual machine allows inbound traffic on the necessary ports we'll install Docker engine and so I'll use the script located at get dod.com I'm going to continue to use the docker client running on my laptop but connect to Docker engine running on the virtual machine using this Docker host environment variable so we'll connect over SSH and have the client running locally but Docker engine running remotely we'll initialize the Swarm we'll modify our Docker compose file to add health checks uh this allows swarm to periodically ping our service and make sure it is healthy and if not replace it or restart it we're going to add a deployment strategy which allows us to deploy with zero downtime then we're going to add support for secrets so that we can handle uh the credentials for that database properly finally we're going to build and push our container images to dockerhub previously we were using Docker comp osed to also build our images for production based setup we want to have a specific image tag that we're running and so I'll build and push those and then finally we'll deploy that stack onto the virtual machine and see it it it running now at this point I also want to issue a slight disclaimer about the way we've configured the database here we're going to be deploying our database alongside the applications into our Docker swarm cluster now that's perfectly fine to do and we do have a Docker volume to persist the data such that it will uh exist Beyond container restarts however we're not implementing backups or anything like that and so if you have an application where your user data is very important you'll want to either Implement a more robust setup that includes periodic backups and the ability to restore from those or move the database to something like a database as a service from one of the cloud providers Amazon has their RDS service Google has Cloud SQL Etc okay so I'm just going to create a virtual machine here on SEO you can use whatever cloud provider you prefer I'll call it Docker course swarm we want one instance I'm going to choose the small machine type I'll use auntu uh let's use the auntu initial user uh we can use the default Network um we do want a public IP address uh we will create a new firewall and we'll use the SSH key that I have already uploaded to my CFO account I'll click create that will take a couple of minutes to spin up uh let's look at the firewall here so right now it's using uh default default I just want to make sure that I have inbound traffic allowed on Port 80 uh so I have this Docker test firewall that I created uh when setting things up let's look at the the rules uh so this is allowing inbound traffic on all ports let's actually turn that off uh we do want to be able to ssh in on Port 22 uh we're going to have web traffic coming in on either Port 80 or 443 uh let's also open up access for our two API ports uh so in this case it would be Port 3000 and Port 880 for the goang API just so we can access those directly now let's go to our VM and attach this firewall to it so instead of default default I want to attach it to Docker test uh and now I want to install Docker engine on this machine so I will SSH into it uh so we created that Ubuntu user it should be using my SSH key that I already have set up the first time connecting you have to accept the fingerprint and now we are connected to that machine if I just try to run Docker it doesn't exist however I know that there's this install script for Docker engine here at get. do.com so I'm just going to do curl on that pipe it to Shell and this will go off and it has conditions within it to install for Ubuntu or Debian and so this is a Ubuntu machine and it should install properly and so Docker is installed but if I try to do something like Docker PS I'm unable to connect to the docker socket I need to change the permit of that socket to be accessible by my current user and so I can do that with the CH own command and now if I do Docker PS I'm able to access Docker so I'm going to exit my SSH session uh however I can still connect to that remote Docker engine by using by using the docker host envirment variable so I'll export Docker host uh and in this case it will be Ubuntu at at that IP address and now even though I'm running this command Docker PS on my laptop it's actually talking to the API of that remote Docker demon we want to enable swarm mode on that remote VM so that's as easy as doing Docker swarm in it and we now have swarm running on that remote host next up we want to take our Docker compose file that we created in in module 8 and update it such that it is ready to go with swarm so I'll snag that Docker compose file copy it over into module 12 and so what are the modifications that we want to make here uh we could just run this directly like I was saying however there's a number of improvements that we want to make so for a production based system uh and let me actually rename it instead of Docker compose let's call it Docker swarm am uh there's a number of things we're going to do first of all we don't need our V based front end because that was a development server anyways so we'll just get rid of that entirely uh next up we're not going to be managing uh the building of our container images via swarm uh so I can get rid of this build section and instead I created uh repositories on dockerhub for each of my images and I'm going to build and push those images to that and so we can see here I've got my Docker course API goang API node and client react engine X in order to build and push my images to those I have this make Target uh that is going to navigate to module 6 and then build version 5 9 and8 of the client node and goang uh Services respectively and so that's effectively going to go into here into each of these directories and execute the build and and push and make targets and so if I do that uh this image that I'll be using for the front end will be Sid palala devops directive Docker course client react enginex version 5 the API node once again will clear out the build portion uh and we'll specify to use the image and this will be Docker course API node 9 was our final uh Docker file and then the final one uh API API API goang will be goang number eight uh clear out the build uh we're going to take inbound traffic on Port 80 and forward that to 8080 CU that was that unprivileged engine X container running on 8080 uh we'll take traffic from 3,000 forward to 3,000 880 on the host go to 880 in the Goen container let's run this config file using Docker swarm on that remote host we already initialized the Swarm uh in this case I need to update this IP address so this was the IP address of the virtual machine I used when testing now I'm going to set it to the IP address of the one I just created uh we've already initialized I'm going to use this uh Docker stack deploy command uh and then pass it the name of my uh swarm file so I'll do make Docker F swarm deploy stack uh because I didn't specify a composed file version in that file I actually need to do that and so at the top here I'll specify that this is version 3.7 let's try it again uh and it's warning me that the docker compose option restart is not supported by Docker swarm uh and so I'm just going to remove those it will handle restarting our services slightly differently uh and the great thing about something like doer swarm is it's declarative so now I can just redeploy the same stack over the top and it will reconfigure my services accordingly now if I do Docker PS we can see the four containers that it's spinning up uh and let me try to access them uh just at that IP address so the IP address was 212 all right and so I now have my containers running remotely on that virtual machine using Docker SW however there's still some improvements that we need to make right I talked about handling of credentials here I'm still passing in those credentials via this environment variable so we need to fix that uh by using a feature called secrets and so generally you would create these secrets via some other process so you wouldn't Define it in your config directly because you don't want to store those credentials with in your get code base and so I have a make Target that we can use to create those Secrets um it essentially uses the docker secret create command uh pass it a name of the secret and then here I'm echoing in the string that will get stored within that secret uh so if I do Docker secret list there's no secrets currently but then if I run this make Target and run it again Docker see secret LS we now have these encrypted Secrets living on that virtual machine that contain the information that we care about so food barbaz as our password uh and then the URL string containing that information as well now in order to consume those Secrets uh we're going to add another field here at the bottom called Secrets uh the first one is database URL and we're going to going to say external true and what external true is saying is that we created this secret already and Docker swarm doesn't need to manage it as a part of this stack we'll do the same for postgres password external true great and so now that we've told Docker swarm about those secrets we need to tell the services that are consuming them to do so and so what I'll do here is we can tell this database service that we want to consume the postgres password uh we no longer need the postgres password here I will change it I will specify the postgres user to be be be postgres so the way that Secrets get consumed is is as a file mounted into the file system um using a a temporary f a temporary Mount structure where that file only exists in memory and so I needed to adjust my application slightly whereas previously they were expecting to read those values from an environment variable now I'm expecting to read them from a file and so I need to tell the application where that file lives and so in this case uh that file will get mounted in to to this location basically under SL run/ secrets and then the name of the secret and so I'm going to tell my application this is where you should go look and read in that that password from the file uh similarly we will have for our apis something very similar so we're telling it we're telling it the path to the file where that Secret's going to be mounted in uh in this case this will be database URL and there we go and then let me just show you real quick the modification that I made to the application so that it can either read those sensitive read those Secrets as environment variables or as files so if we go to our node.js application here Source database and so when I'm populating this database URL variable I either load it from the environment or if that doesn't exist then I read from the file system at this path that I'm passing it very similar on the goang side where when I'm initializing I either get the database URL directly or if it's empty then I'll read the file and use that instead and so now I can redeploy using those secrets now also it takes my swarm config file and turns that into a stack with many services in it so I can do make swarm LS and that's going of list the different services and we see that we have our different replicas here and they all appear to be running except for the goang one let's reload our application and so it looks like our front end is still working but our apis may not be so if I navigate to and so let's take a look at the logs for those containers to see what might be going on let's look at the node API logs uh it looks like we are refusing to connect So within the the node container I can print the environment I can see that we have this database URL file that looks good um what happens if I C that file out let's see uh okay so it looks like within so it looks like when I tried to use this dashn option for the echo command to avoid new lines it actually included that in my secret file which is what I didn't want to happen and so I'm actually going to remove my stack make swarm remove stack so it looks like the dashn I was using to remove the new line at the end of the file was actually getting included in the secret so I'm just going to use print F instead do make create Secrets uh and now let me redeploy my stack let me uh make sure that the secret looks okay within those within those containers uh so in this case let's cat this out within the postgres container that looks good and then within let's say the goang container or no we'll do it within the node container because the goang wouldn't have cat within [Applause] it okay so that looks much better and now do my services look healthy uh they do and if I go to my IP address address and refresh I can now access my application again okay so minor dour there uh how I was creating those Secrets there was an issue with the that dashn was making its way into the secret itself and so I fixed that by using print F and now the secrets appear to be working properly now the next thing that I wanted to do is to add a an ability to deploy new versions without bringing down the old version first and so for that we actually use the deploy configuration option and so for our front end we will add a deploy section and within this section there's a few things We'll add we'll put it on replicated mode with a single replica so we could change this to 10 and it would spin up 10 copies of our front end but we just need one for now uh and then the key configuration here is update config we want to start the new container before we terminate the old container so that's what this order says you can actually do stop first if you want to do the reverse and so this is how you this deploy configuration is how you tell Docker swarm how you want to do rolling updates uh we'll have pretty much the same configuration for our other services as well uh let's make the goang application have two replicas just to show how easy that is to do and then the final piece of the puzzle that we wanted to add is health checks uh and so we want to have a way for Docker swarm to know whether our containers ready or not so that when it's checking should I bring down the old container is the new one ready to accept traffic the way that we do that is via health checks now the health check config uh is relatively straightforward you use the health check key you give it a test and in this case we're saying run this command inside of that container so it's going to try and curl uh Local Host 8080 and then I've configured my engine X configuration so that when you hit uh the slping route it will just return pong you then tell it how frequently you want to issue this health check how long you want to wait before you consider it to be timed out out and how many tries before you consider it to be a failure finally the start period is how long you expect it to take before the container is ready from initial boot up uh before it needs to do this test in this case our container image already had curl built in so there was no additional uh changes to the application that we needed to make for the API node service we'll have a very similar health check I'll add it right here um in this case though we're not going to be we don't have curl installed into our container image we wanted to keep our container image as minimal as possible we didn't want to have extra utilities floating around and so in this case I actually defined a health check in JavaScript that's going to be run from within the container and tell us if the container is healthy or not I'll show you what that looks like so under API node under health check uh uh we're using the HTTP package uh we say take it up to 2 seconds to Ping the slping route on Port 300 if we get back a 200 code great if we don't then fail and so this is what is getting run by that health check inside the container every 30 seconds for goang it will be almost identical uh in this case I have a separate binary that gets compiled in my container image and so if I look at the application I have this health check package within here again I'm just pinging Local Host on the route ping I have that defined in my application to return a 200 and then in my Docker file when I'm building my application I build both my health check as well as my main.go and so these are two separate binaries that live in my container image one of which is the application the other one gets run by this health check okay so with those health checks defined uh I can now redeploy my swarm stack and now I can list out those Services we'll see something interesting though if I do a Docker PS we now see under the status field see that these have status of starting meaning that it's using that health check to determine whether or not the container's healthy or not and until it is it won't Route traffic to those containers and it keeps the previous containers alive such that they can continue to receive traffic now that those health checks have started passing we can see that they've been up and they are now healthy and so we should be able to continue to access our application with no downtime and so now whenever we make a change we would build and push that container image update our Docker swarm configuration if needed redeploy and drer swarm would be intelligent enough to use those health checks to determine when we can Route traffic to the new containers or not you'll also see that because I have two replicas specified in my config we get two containers both running that goang image a couple of other additions that we could do uh we could configure our engine X front end to retrieve a certificate from let encrypt for example and store that in a volume such that it can persist across container restarts because right now we're serving traffic on Port 80 over HTTP unencrypted uh if we go to https we're not serving on 443 so that doesn't exist uh we also can update our DNS settings for a domain of our choosing and so for example we can go to cloudflare uh under devops directive I already have the setup but I'll update it to be the new IP address and so we can update this and that will take a few minutes to propagate but because I already had the old one set up we can just navigate to it directly and as you can see we're now routing traffic from that domain to our virtual machine hitting our application running within darker form and everything's behaving as expected you'll also notice that here I do have https and now this is a little this is a bit of a hack because it's using Cloud flares flexible TLS and so I have this set up such that encrypted traffic goes to cloudflare that's where it terminates TLS and then it sends that request unencrypted to the origin server and so if you're not dealing with particularly sensitive information depending on your threat model this may be sufficient however if you are dealing with sensitive information or there's requirements about the data that you're handling you would want to terminate TLS on your system uh this can be done like I was saying you could have engine X go off and get a let's encrypt certificate uh at its first startup uh you also can you can also handle that kind of at the Ingress layer for your doer swarmm cluster uh using something like traffic or another Ingress controller that can handle that and so with that we have our application deployed to a virtual machine running in the cloud being managed by Docker swarm we've set it up to use secrets to handle the credentials properly use health checks to allow us to have zero uh downtime deployments uh and so that's great for a single node if you want to add High availability to your system or or deploy to multiple nodes it starts to get a bit more complex especially around how it manages volume so in this case we have a single volume for our postgres data on that virtual machine if we wanted to have a multi-node cluster we would need to look at volume plugins and how to manage those how to allow Docker swarm to manage those across multiple hosts and so I didn't want to go into that level of depth in this course if you do want to dive a bit deeper on Docker swarm there's a great course from Brett fiser called Docker swarm Mastery that I would suggest you take a look at that'll be a full Deep dive on how to use swarm there's a recent reinvigoration of excitement around swarm They just added support for uh what's known as the CSI container storage interface which is a standard interface for managing persistent storage that is used in kubernetes has a lot of usage and maturity from that domain that's now going to get pulled hopefully back into swarm and make it a viable option for production deployments moving forward as I mentioned this is just one of many many different options for deploying containers um hopefully this gives you some ideas but do be on the lookout for the bonus content around uh how to use something like Railway or kubernetes to deploy the same sample application and with that we've reached the end of the course congratulations hopefully you feel ready to go off and incorporate containers into your software development and deployment approaches and use the lessons you've learned to improve the software systems within your organization to recap everything that we've covered we started with some fundamentals exploring why we should even care about containers in the first place how the evolution of virtualization brought us the current state of containers the underlying Technologies within Linux that enable containers to work and how Docker fits into the picture with that as our base we were then able to move on to learn how to use containers we used public third party containers for things like databases and interactive test environments we built out a multi service web application containerized it established a nice development workflow and then finally deployed our application to the cloud I hope that you enjoyed taking this course as much as I enjoyed making it also if you'd like to join a community of other devops practitioners I run a Discord server with lots of amazing folks where we talk about this type of thing all the time and answer each other's questions there's a link in the description if you'd like to join if you want more devops and Docker content YouTube should be recommending another video over there that's it for this course and remember just keep building

hey hey hey do you want to see me build a docker container yeah let's do it real quick [Music] done want to see it again done man that was fast [Music] containers if you're in it you got to know it if you want to learn cloud azure aws google linode devops networking systems i think i said everything right so i hope you have your coffee ready in this video we're covering three things first what the junk is docker two you're getting hands-on with docker i'm giving you access to a free lab thanks to the sponsor of this video lenode and then three we'll go a bit deeper talk about the big why why docker alright let's do it okay what the junk is a docker container whoa whoa whoa we're getting ahead of ourselves here before we can define what a docker container is we have to first define what it's replacing virtual machines what is a virtual machine how many of you already know check the time stamps to jump forward but we're going to cover what it is real quick and i actually have a ton of virtual machines running on servers in my data center right now and i've been meaning to get those servers out of there because i'm paying a high bill i want to bring those home with me right now let's go get it [Music] whose things should i unplug goodbye davidson [Music] [Applause] so here's one of my babies ugh ugh ugh my servers now before we answer the question what in the world is a docker container we have to first talk about what's a virtual machine let's dig in so this is my server traditionally before virtual machines you would install one operating system on this piece of hardware oh bit dusty all of the server's resources from the cpu to the ram to the hard drive all of it was dedicated to one operating system one server os so like i might just have linux and this is my linux server or i might just have windows this will be my windows server 2016. now what's the issue with that well none really unless you want another server another operating system so before virtual machines if you wanted a windows server and a linux server you had to have two physical servers like this so that's where virtualization comes in instead of installing one operating system we're going to install something called a hypervisor [Applause] now a hypervisor is kind of like an operating system but it only has one goal its goal is to help you carve up or divide your server's resources into multiple servers now there are a ton of hypervisors but the most popular one would be vmware which is what i have installed on my servers and specifically it's called esxi bonus points if you tell me in the comments what this means or what it stands for and simply what it allows us to do is take a portion of our server's resources maybe two gigs of ram two cpus 100 gigabytes of hardware drive space and this will be our ubuntu server and then on the same server i could say over here i could have one cpu not cup one cpu two gigs of ram 40 gigs of hard drive space and that will be my windows server 2016 and i can keep doing this until my server runs out of resources or i can even make these bigger going from i don't know one cpu to 41 i mean you can go crazy virtual machines are amazing and people use them every single day but that's not how docker works in fact docker's kind of well not just kind of it's different when we use docker containers it takes us back to before virtualization we're going to install one operating system on this entire thing so one operating system gets access to everything our server has to offer but why do that that's the big question right what is docker doing here here's what it is and this is crazy i want to drop this line on you okay where virtualization and virtual machines are virtualizing hardware docker virtualizes the operating system what what does that even mean [Applause] [Music] now this right here is what we commonly see when we visualize how virtualization works we have our hardware we've got our our hypervisor and then we have our guest operating systems but let me show you what docker looks like we'll always have our hardware but instead of a hypervisor like we just discussed we're going to have one operating system installed on top of this hardware or onto the hardware and let's just say for our case we are going to install ubuntu now here we go we'll then have our good old buddy docker fly in and uh make some magic happen docker installs like any other application you might install on linux it'll be a process running or a daemon running and then using docker we will virtualize portions of our operating system in this case ubuntu to create containers which by the way we'll define here in a moment but they're really doing the same thing as these virtual machines over here they have the same function and they have a lot of the same qualities which is why it's kind of cool how they can replace virtual machines controversial but that's what i think and these containers much like vms can be running a different operating system centos and even ubuntu you can install ubuntu on ubuntu what okay but what are these things what are containers i like to think of them as crazy fast lightweight micro computers and don't let micro fool you like they're they can be as powerful as you want them to be and similar to a virtual machine they'll have their own operating system their own cpu processes their own memory their own network and the most important thing and i think it's really appropriate for nowadays is that they're quarantined they are isolated they are secure as far as they understand they are their own thing they might as well be installed on bare metal servers because that's how isolated they are wilson they're as isolated as you are right now what too soon let me say it again these suckers are lightweight and they are wicked fast like you saw at the beginning of this video click bam it's there now we'll cover why they're more lightweight why they're faster and and more of why they're amazing here in a moment but i think it's time i think it's time you got your hands dirty your feet wet all those things we're gonna do that right now time to get your hands on with docker containers i'm so excited about this you might say i can't contain myself sorry okay here we go fire up your web browser go to the link below lynnode.com forward slash network chuck and you get 20 of free credit more than enough to play with docker so go ahead and create that free account and we will start labbing right now now as i mentioned before lenode is a cloud service provider very similar to aws azure google and we're going to use them as our cloud service provider right now so from the dashboard we'll go to the top right and click on create and we'll create a lenode now right now you should be on distributions i want you to click on one click click on that and this is what's really cool about this look at all these pre-built apps you can just deploy in the cloud wordpress openvpn run your own vpn server in the cloud minecraft okay cool csgo i kinda wanna do this now anyways the one we wanna look at now is obviously docker bam a select docker now you can install and run docker anywhere windows linux you can install on a virtual machine but this is great because it makes it easy docker is already ready to go i'll just click create i'll select my size 10 bucks a month hey that's half of what you are getting for free right now i'll name it my friggin sweet doctor lab and then a root password for your root user and then click create ah forgot my region i'm in dallas that way i get super fast performance go now this is essentially creating a virtual machine for you in a lino cloud and it's pre-baked with docker and once yours is ready click on launch console at the very top here and right here in the browser you have access as soon as it's done doing your its updates but if you're like me i like to use your own terminal more of a homey feel so i'll grab the public ip address right here copy that and open up my own which is solar putty i highly recommend solar putty you should use it go download it now link below it's free and i'll connect your name root and the password is what i just set and i'm in i'll clear this out i've got my own ready to run docker empire yeah i'll call it that so for this lab we're going to do three things we're going to install centos we're going to install alpine and a third mystery thing which makes docker the coolest thing ever so hang tight but let's do let's do centos first first i need the image so i'll do docker pull centos what this is doing is it's going out to the docker hub which is a registry of docker images these images are what we use to run or create our containers i'll show you real quick so we'll go ahead and pull the image for centos it'll go out and find it and download it bam we've got it so now with that centos image we're going to run our first docker container so it'll be the command docker run we're gonna use a few switches dash d and dash t then we'll do dash dash name to name this container i'll name mine can't contain myself name yours whatever you want and then at the end i'll specify the image i want to use which is the one we just pulled down sent os and this is how fast docker containers are created and run it's running that was it a virtual machine booting a virtual machine takes a whole lot longer than that that's crazy so let's make sure it's running we'll use command docker ps to see our running docker containers there she is and she's so pretty look at her that is centos isolated secure running with its own operating system cpu memory network oh it's quarantined but let's um let's walk into the room real quick we can connect to centos and start playing around with it by using the command docker exec i'll use the switch dash i t and then i'll reference the name we named it so it was can't contain myself for me if i spell it right and then after the name i'll specify i want to connect to the bash shell boom and i'm in centos i am in a container and look i mean it even has its own file system like it's completely separate to get out of there we'll just do a quick type and exit and we're back home and you'll know where you are if you pay attention to root at blah blah blah that's the container id we were just in centos over here is my docker machine here so i'm going to clear my screen let's do another one we'll do docker pull let's say we pull down alpine a very popular linux distribution because it's very uh very lightweight especially for docker see i mean it pulled it down so quickly and we'll do docker run dash t dash d i'll name this oh yeah and then the image name was alpine boom another one started if i do docker ps ps ps two containers running awesome let's connect to it docker exec dash it oh yeah and then alpine i'll just connect to the shell so i'll type in sh and boom i'm there i'm an alpine a separate operating system a separate container quarantined it's so cool you know what you can do this too if you want i'm gonna install a whole bunch of things right now okay so now if i do docker ps i got a few things going on i've got three different flavors of ubuntu running right now 16 14 and 18 on the same server that's amazing wordpress it's it's it's cool now for number three and i want you to do this with me this is actually a um a docker image that i created a while back for my azure course on cbt nuggets if you want to check the course out link below but using this image i want to demonstrate the power the amazingness is that a word yes it is now of docker let's do it real quick so first i want you to pull down my image now to find it let's go to docker hub it's hub.docker.com this is a registry where you can download and pull all these images from docker and they have a ton of stuff um some of the stuff they have is from me so if you type in network chuck in the search bar i come up here and actually have two out there i don't know if the other one's for but the one i want you to click on is coffee based training has 879 downloads that's pretty cool jump in there and even shows you like the command you can use to pull down this image now don't use that just yet i want you to copy the network chuck forward slash nc coffee and we're going to use the tag french press and i'll show you how to use a tag right now so the command will be docker pool i'll paste in my name here the network chucky chucky the network chuck forward slash nc coffee and then colon french press the tag basically gives us the ability to download different versions of a particular image i did that with ubuntu just earlier i could specify the tag 16 17 whatever and download different versions so i'm going to pull down my image boom it's ready now i'm going to run it clear things out here so i'll do docker run run run dash t dash d and then one more switch here dash p which sends reports and we're going to map port 80 which is a port for our website obviously we'll map that from our docker container to our host and for any exam people out there who are studying for the devnet associate or ccna the port right here at the end is for the docker container and the port right here is for the host so i'm essentially saying match port 80 to port 80. then i'll name my guy name it nc coffee and then i'll specify the image which is the network chuck forward slash nc coffee and then colon french press and um it'll be quick done i'll do docker ps it's running now here's what i want you to do if you go back to your linode dashboard here grab the ip address once more copy it and then just plug that into your browser go to it it's a docker container running my website i made obviously it's a professionally designed website click on the link it'll take you to cbt nuggets i mean it's it's top notch if you want to hire me for web development i'm available now let me ask you a question did you set up this website did you alter any settings install any programs did you put in this like did you do any of this no but if you pulled down that image and ran that container like i just demonstrated you have a website that looks just like this you see that container not only has its own os cpu processes memory but it also has all the settings i applied my website configuration all the prerequisites all the dependencies packaged together and a neat little package a container no need to mess with it it's perfect and you can move that sucker to any computer any linux based machine and it runs like a champ i can move this same docker container to aws to azure to your computer to my computer to wherever and it's going to run the same and that's the power of docker so everything from the os which is debian linux to the things i had to install to make it work nginx which is a web server software and then all the website files that i used which you can tell by the quality of the site took a lot of stuff and then anything else i might need all dependencies packaged into a nice neat container that can run anywhere and now we're at number three of our objective kind of talking about the big why why why docker containers why are they so killer compared to virtual machines and really why should you care about them we've seen a few of the things like they're lightweight and they're fast and we just saw what i think is one of the coolest things about it is that they're portable no matter where they run they run another example of why this is absolutely killer is like look here i've got different flavors of ubuntu i might have apps that aren't compatible to run on a certain version of ubuntu oh and by the way before we go any further if you want to stop any of these containers you can do that here just do stuff docker stop and then the container name so i'll drop like one of my uh i'll drop alpine oh yeah and then i can start him right back up again docker start oh yeah and bam he's up again and if you want to like see how much cpu they're using memory network we can do docker stats and we get this cool i mean i think it's cool view of all that's going on real time and then ctrl c to back out of there well this is for all of you who told me to try out arch linux i can install as a docker container as well and boom just like that i'm running arch linux okay at this point we've learned a bit about docker containers we've even created a few and played around with the command line that that was fun now we're gonna go just a little bit deeper not too deep because we we don't have the time but we'll talk about okay why is it so fast why is it so lightweight and how is the industry using it right now this will be real quick let's talk about it so first of all why is it so fast how does it boot up so fast how do we create containers so quickly what's the voodoo behind that the short answer is that you only need one kernel notice that over here when i'm demonstrating docker all of these os's are linux based they all share the underlying linux kernel just like ubuntu does so ubuntu when we installed him he's running off the linux kernel and it's the same linux kernel that all of these os's right here would use so this part of the os is already up and running and that's in stark contrast over here on a virtual machine you see each of these os's have their own linux kernel they're spinning up i'll just put lk for the rest because they are virtual machines and every time you deploy a new virtual machine that's another linux kernel and then the operating system and that steps into the other part is they're lightweight why are they lightweight because we don't need more than one linux kernel you see these virtual machines to run more than one os and to have this app isolation like we want to with containers we have to have a guest operating system every time which involves another kernel it involves a hypervisor which is also kind of heavy so it's overall just heavy there are more parts and pieces that have to be running to make that work whereas a container is much more simple lightweight and fast now one thing i want you to note and you probably notice this i have a windows server over here but i didn't install windows server over here on linux you can't do that containers are available docker you can install on windows based systems windows 10 windows server i believe 2016 and up but but but you can only install windows based containers on a windows system same goes for linux you can only install linux based containers on a linux system because they share the same underlying os or kernel and yeah docker on windows is cool you can install docker containers on windows and it shares the same qualities as a linux docker container they're isolated they have their own processes and memory and network it's just as cool now what i've given here was a very high level overview we didn't go very deep at all docker is a big thing like there's there's an entire docker certification it's it's huge and if you want to learn more go deeper because there's a lot more maybe you want to learn more about how containers work maybe you want to learn hey like they're using control groups which are defining how much cpu how much memory how much os resources that you're giving each docker container and then how they're segmenting it isolates isolating everything with namespaces and by the way these two concepts that can be attributed to containers have been around for a long time containers have been a thing for a very very long time docker isn't the only type of one either so it's a whole big thing so if you want to dive deeper if you want to create your own docker containers there's a great course from sean powers at cbt nuggets link below and uh if you want to learn more just about the the behind the scenes and under the hood uh i'll link some books and stuff below as well now the last thing i wanted to mention is how is the industry using docker well we already demonstrated that we could put a docker container up in the cloud linode we just did that you can move that same docker container to azure to aws they have the same sort of situation with docker it's just like a web app you deploy and you can launch your containers this is killer for developers because they can write their code deploy it in a docker container and know that it works everywhere and that's a problem with developers that they've had for a long time is that they'll write an app on their computer and it works great they spent time polishing it you know dotting their eyes and all kinds of stuff and that same code is run by i don't know a tester and it doesn't work and the developer's like well hey it worked on my machine i don't know what you're doing no more of that junk no more of that because if you deploy in a docker container everything that app needs is right there in its container it's contained it's isolated and finally the last thing that has kind of taken the industry by storm is the concept of micro services this is more for developers but as a person who's an i.t and as you move into more devops stuff this is stuff you'll understand and have to work with microservices involves taking portions of your application um your stack and uh and segmenting it into smaller bits and pieces so traditionally you might have you know one server where you install your entire application everything you need so for example wordpress you might install your wordpress server and your your mysql sql database on the same box and that's traditionally what developers might have done is they would install as many compatible apps as they could on one server but now with containers they can take one app in particular like maybe just the wordpress server and have that in one container and then maybe take the mysql portion and have that in one container so they're isolated there's no conflicting things and when you want to update one portion of your app you're just updating one container and that's a whole other topic for a discussion a whole big world it's fun it's exciting and i'm hoping that this video got you started on docker and it won't be your end point i hope you go further with this and you know let me know if you want me to make more videos about docker oh well that's about it docker's a big thing and i hope you uh be a fun watching this video let me know what you thought in the comments and also let me know how you're going to use docker i mean there's a variety of ways like for example in my own home i have docker running on my synology server and i have a container that's running home assistant which is a home automation um framework i guess it's really cool you can also run things like pie hole and just a ton of stuff it's it's amazing what you can do not to mention that if you're learning linux or you want to try out all kinds of different linux distributions you saw how easy it was to set up that lab oh my goodness it's so powerful whew i've been talking too much so if you haven't already hit that subscribe button if you like what i'm doing here and hit that notification bell to always get notifications when i go live or when i have new videos and also don't forget to support the sponsor of this video linode if you want to deploy a website or an app or just a minecraft server uh link below get 20 credit so if you didn't get a chance to lab do it now i'll catch you guys later [Music]

welcome to the ultimate docker course in this course i'm going to take you on a journey and teach you everything you need to know about docker from the basics to more advanced concepts so by the end of this course you'll be able to use it like a pro as part of your software development workflow if you're looking for a comprehensive and highly practical course that takes you from zero to hero this is the docker course for you we're gonna start off with a really simple project so you understand the basics then we'll use docker to run and deploy a full stack application with a front end back end and a database so you learn all the necessary techniques and apply them to your own projects i'm ash hamadani and i've taught millions of people how to advance their software engineering skills through my youtube channel and online onlineschoolcodewithmash.com if you're new here be sure to subscribe as we upload new videos all the time now let's jump in and get started [Music] let's quickly talk about what you need to know to take this course to take this course you don't need any prior knowledge of docker because i'm going to teach you everything from scratch but you should have at least three months of programming experience ideally you should have built at least one application so you should know concepts like front-end back-end api and database it doesn't matter what languages and frameworks or what database engines you're familiar with but you should know what these concepts are all about also you should be familiar with basic git commands like cloning a github repository committing code pushing and pulling just the basics nothing more with all that let's move on to the next lesson [Music] now we all have different ways of learning things but let me tell you what i think is the best way to take this course this course is highly practical so you should be active while watching this course in my opinion it's best if you watch each lesson take some notes you can either take notes on a piece of paper or using your favorite note-taking tool just write some keywords some keywords that help you remember what we talked about then after each lesson go through your notes and repeat the same steps i have taken in that lesson so if i talked about a bunch of commands play with those commands make sure you understand how they work if you follow that i promise you by the end of this course you're gonna master docker and use it like a pro alright our journey to master docker begins here in every section you're going to discover something new about docker in this section we're going to talk about what docker is and why it's so popular then we're going to talk about virtual machines and containers next we're going to talk about the architecture of docker so you understand how it works then we're going to install docker and get our hands dirty i'm going to give you an overview of your development workflow with docker and then we're going to see that in action using a really simple example so by the end of this section you will have an idea what docker is all about i'm so excited about this section i hope you are too so let's jump in and get started [Music] so what is docker and why is everyone using it these days well docker is a platform for building running and shipping applications in a consistent manner so if your application works on your development machine it can run and function the same way on other machines if you have been developing software for a while you've probably come across this situation where your application works on your development machine but doesn't somewhere else can you think of three reasons why this happens well this can happen if one or more files are not included as part of your deployment so your application is not completely deployed it's missing something this can also happen if the target machine is running a different version of some software that your application needs let's say your application needs node version 14 but the target machine is running node version 9. this can also happen if the configuration settings like environment variables are different across these machines and this is where docker comes to the rescue with docker we can easily package up our application with everything it needs and run it anywhere on any machine with docker so if your application needs a given version of node and mongodb all of these will be included in your applications package now you can take this package and run it on any machine that runs docker so if it works on your development machine it's definitely going to work on your test and production machines now there's more if someone joins your team they don't have to spend half a day or so setting up a new machine to run your application they don't have to install and configure all these dependencies they simply tell docker to bring up your application and docker itself will automatically download and run these dependencies inside an isolated environment called a container and this is the beauty of docker this isolated environment allows multiple applications use different versions of some software side by side so one application may use node version 14 another application may use node version 9. both these applications can run side by side on the same machine without messing with each other so this is how docker allows us to consistently run an application on different machines now there is one more benefit here when we're done with this application and don't want to work on it anymore we can remove the application and all its dependencies in one go without docker as we work on different projects our development machine gets cluttered with so many libraries and tools that are used by different applications and then after a while we don't know if we can remove one or more of these tools because we're always afraid that we would mess up with some application with docker we don't have to worry about this because each application runs with its dependencies inside an isolated environment we can safely remove an application with all its dependencies to clean up our machine isn't that great so in a nutshell docker helps us consistently build run and ship our applications and that's why a lot of employers are looking for people with darker skills these days so if you're pursuing a job as a software or devops engineer i highly encourage you to learn docker and learn it well and that's exactly what this course is all about i'm going to take you on a journey and teach you everything you need to know about docker so you can use it like a pro no copy pasting commands here [Music] so in the last lesson i briefly talked about containers a container is an isolated environment for running an application now one of the questions that often comes up is how are containers different from virtual machines or vms do you know the differences well a virtual machine as the name implies is an abstraction of a machine or physical hardware so we can run several virtual machines on a real physical machine for example we can have a mac and on this mac we can run two virtual machines one running windows the other running linux how do we do that using a tool called hypervisor i know it's one of those computer science names in simple terms a hypervisor is software we use to create and manage virtual machines there are many hypervisors available out there like virtualbox and vmware which are cross-platform so they can run on windows mac os and linux and hyper-v which is only for windows so with a hypervisor we can manage virtual machines now what is the benefit of building virtual machines well for us software developers we can run an application in isolation inside a virtual machine so on the same physical machine we can have two different virtual machines each running a completely different application and each application has the exact dependencies it needs so application 1 may use node version 14 and mongodb version 4 while application 2 may use node version 9 and mongodb version 3. all these are running on the same machine but in different isolated environments that's one of the benefits of virtual machines but there are a number of problems with this model each virtual machine needs a full copy of an operating system that needs to be licensed patched and monitored and that's why these virtual machines are slow to start because the entire operating system has to be loaded just like starting your computer another problem is that these virtual machines are resource intensive because each virtual machine takes a slice of the actual physical hardware resources like cpu memory and disk space so if you have 8 gigabytes of memory that memory has to be divided between different virtual machines of course we can decide how much memory to allocate to each virtual machine but at the end of the day we have a limit in terms of the number of vms we can run on a machine usually a handful otherwise we're going to run out of hardware resources now let's talk about containers containers give us the same kind of isolation so we can run multiple applications in isolation but they're more lightweight they don't need a full operating system in fact all containers on a single machine share the operating system of the host so that means we need to license patch and monitor a single operating system also because the operating system has already started on the host a container can start up pretty quickly usually in a second sometimes less and also these containers don't need a slice of the hardware resources on the host so we don't need to give them a specific number of cpu cores or a slice of memory or disk space so on a single host we can run tens or even hundreds of containers side by side so these are the differences between containers and virtual machines [Music] let's talk about the architecture of docker so you understand how it works docker uses a client server architecture so it has a client component that talks to a server component using a restful api api api the server also called the docker engine sits on the background and takes care of building and running docker containers but technically a container is just a process like other processes running on your computer but it's a special kind of process which we're going to talk about soon now as i told you unlike virtual machines containers don't contain a full-blown operating system instead all containers on a host share the operating system of the host now more accurately all these containers share the kernel of the host what's a kernel a kernel is the core of an operating system it's like the engine of a car it's the part that manages all applications as well as hardware resources like memory and cpu every operating system has its own kernel or engine and these kernels have different apis that's why we cannot run a windows application on linux because under the hood this application needs to talk to the kernel of the underlying operating system okay so that means on a linux machine we can only run linux containers because these containers need linux on a windows machine however we can run both windows and linux containers because windows 10 is now shipped with a custom built linux kernel this is in addition to the windows kernel that's always been in windows it's not a replacement so with this linux kernel now we can run linux applications natively on windows so on windows we can run both linux and windows containers our windows containers share the windows kernel and our linux containers share the linux kernel okay now what about mac os well mac os has its own kernel which is different from linux and windows kernels and this kernel does not have native support for continuous applications so docker on mac uses a lightweight linux virtual machine to run linux containers all right enough about architecture next we're going to install docker and that's where the fun begins [Music] all right now let's install the latest version of docker if you have an existing version of docker on your machine i highly encourage you to upgrade to the latest version because your version might be old and not compatible with the version i'm using in this course so here i'm using docker version 20.10.5 okay so to get docker go to this page docs.docker.com get docker or you can just google get docker or install docker now on this page you can see instructions for downloading and installing docker desktop for mac and windows as well as docker engine for linux so on mac and windows we have docker desktop which is the combination of docker engine plus a bunch of other tools at the time of recording this we don't have docker desktop for linux we only have the docker engine but of course that might change in the future so a couple of notes for my mac and windows users let's go to this page so over here you can download the latest version from docker hub but before doing this i highly encourage you to go through a system requirement and make sure your computer satisfies these requirements otherwise you might encounter weird issues so the installation is pretty straightforward when you go to this page you're going to download a dmg file just drag and drop this onto the applications folder and then start it this is very important a lot of people miss that step so when you start docker engine by double clicking on this application you're going to see the docker icon on the top status bar if you don't see this docker engine is not running and there is nothing you can do so you have to wait for this to start the same applies to my windows users so back to this page let's look at the instructions for windows so once again you can download the latest version from docker hub but once again make sure to read system requirements one of the things that is really important is enabling hyper-v and containers windows features so i don't have a windows machine to show you here but just go to the page where you can turn on or turn off windows features there make sure you have enabled hyper-v and containers it's pretty straightforward but if you can't find it just google it i'm pretty sure there are tons of articles out there also at the end of the installation you might get an error saying ws2 installation is incomplete basically what this error is saying is that you need to upgrade the linux kernel that is shipped with your windows so just click on this link this is going to take you to this page on microsoft.com where you can download the latest linux kernel so just click on this link to get an msi file run it and then you have to restart your computer now once your computer is restarted you need to wait a little while until docker engine started depending on your machine this might take several seconds or up to a minute so wait a little while then open up a terminal window and run docker version so over here you can see the version of the client and the server if the server which is docker engine is not running you are not going to see this information and this applies to all operating systems windows mac os and linux so make sure docker engine is running before going forward if you encounter any errors you can post your question on our forum at forum.codewithmosh.com or if you want a faster response just google the error message i'm pretty sure hundreds of people have encountered the same problem as you [Music] now let's talk about your development workflow when using docker so to start off we take an application it doesn't matter what kind of application it is or how it's built we take that application and dockerize it which means we make a small change so that it can be run by docker how we just add a docker file to it a docker file is a plain text file that includes instructions that docker uses to package up this application into an image this image contains everything our application needs to run everything typically a cut down operating system a runtime environment like node or python it also contains application files third-party libraries environment variables and so on so we create a docker file and give it to docker for packaging our application into an image once we have an image we tell docker to start a container using that image so a container as i told you is just a process but it's a special kind of process because it has its own file system which is provided by the image so our application gets loaded inside a container or a process and this is how we run our application locally on our development machine so instead of directly launching the application and running it inside a typical process we tell docker to run it inside a container an isolated environment now here's the beauty of docker once we have this image we can push it to a docker registry like docker hop docker hub to docker is like github to git it's a storage for docker images that anyone can use so once our application image is on docker hub then we can put it on any machines running docker this machine has the same image we have on our development machine which contains a specific version of our application with everything it needs so we can start our application the same way we started it on our development machine we just tell docker to start a container using this image so with docker we no longer need to maintain long complex release documents that have to be precisely followed all the instructions for building an image of an application are written in a docker file with that we can package up our application into an image and run it virtually anywhere this is the beauty of docker next we're going to see this workflow in action [Music] in this design i'm going to walk you through a typical development workflow now don't try to memorize anything in this lesson because i just want you to see the big picture so everything i show you in this lesson we're going to cover in depth later in the course so here in this terminal window i'm currently on my desktop i'm going to create a new directory called hello docker then go inside this directory and open it in visual studio code so i'm going to use vs code as my editor but you can use any editor that you prefer okay now in this directory i'm going to add a new file called app.js you're going to write one line of javascript code you don't have to be a javascript developer you don't even need to learn javascript so just follow along with me so here we're going to write console in lowercase dot log hello so with this we're going to print a message on the terminal let's say this is an application and we want to dockerize this application so we want to build run and ship it using docker so typically without docker if you want to ship this application or more accurately this program to a different computer on that computer we need to install node and then we can go to the terminal and type node app.js so we get the output so here are the instructions for deploying this program we need to start with an operating system then we need to install node which is an execution environment for javascript code next we need to copy our application files and finally we need to run node app.js so we have to follow four steps just for a simple program what if you were working with a really complex application you would end up with a complex release document that had to be precisely followed now this is where docker comes to the rescue we can write these instructions inside a docker file and let docker package up our application so back to vs code we're going to add another file to this project called docker file so capital d and all the other letters are lowercase and this file doesn't have any extensions okay now vs code is asking if you want to install the recommended extensions for docker we can go ahead with that good so back to this docker file here we write instructions for packaging our application so typically we start from a base image this base image has a bunch of files we're going to take those files and add additional files to it this is kind of like inheritance in programming okay so what is the base image well we can start from a linux image and then install node on top of it or we can start from a node image this image is already built on top of linux now how do i know these names well these images are officially published on docker hub so if you go to hub.docker.com and search for node you can see the official node image here so docker hub is a registry for docker images now back to our docker file so we start from a node image now if you look at docker hop you will see that there are multiple node images these node images are built on top of different distributions of linux so linux has different distributions or different flavors used for different purposes now here we can specify a tag using a column to specify which linux distribution we want to use for this demo i'm going to use alpine which is a very small linux distribution so the size of the image that we're going to download and build on top of is going to be very small okay so we start from that image then we need to copy our application or program files for that we use the copy instruction or copy command we're going to copy all the files in the current directory into the app directory into that image so that image has a file system and in that file system we're going to create a directory called app okay now finally we're going to use the command instruction to execute a command what command should we execute here node app.js but this file is inside the app directory so we have to prefix it with the directory name alternatively here we could set the current working directory work dir to slash app and then we don't need to prefix this with the directory name so when we use this instruction all the following instructions assume that we're currently inside the app directory okay so so so these instructions clearly document our deployment process now now now we go to the terminal and tell docker to package up our application so we say docker build we need to give our image a tag a tag to identify so t t t here we specify a name like hello docker and then we need to specify where docker can find a docker file so we're currently inside hello docker directory and our docker file is right here so we use a period to reference the current directory let's go with that now you might be expecting an image file inside the current directory but back in vs code look there is nothing here because the image is not stored here and in fact an image is not a single file how docker stores this image is very complex and we don't have to worry about it so back to the terminal to see all the images on this computer we type docker images or docker image ls which is short for list so take a look on this machine we have a repository called hello docker in this repository we have an image with this tag latest so docker added this by default we'll talk about this later in the course but basically we use these tags for versioning our images so each image can contain a different version of our application okay now each image also has a unique identifier here we can see when the image was created and the size of this image so because we used node from linux alpine we ended up with 112 megabytes of data in this image so this image contains alpine linux node and our application files and the total size is 112 megabytes now if we used a different node image that was based on a different distribution of linux we would end up with a larger image and then when deploying that image we would have to transfer that image from one computer to another so that's why we use node alpine because this is a very small image okay so we have built this image now we can run this image on any computer running docker so on this machine which is my development machine i can say docker run and then type the image name hello docker and it doesn't matter which directory i'm in because this image contains all the files for running our application now look we see the message on the terminal now i can go ahead and publish this image to docker hub so anyone can use this image then i can go on another machine like a test or a production machine and pull and run this image in fact i've done this before recording this video so back to docker hub look i have this repository code with mosh slash hello docker now we can take this image and run it on any computers let me show you something really cool so let's search for play with docker let's go to this page and log in here we need to sign in with our docker id anyone can create this id on docker.com so let's go ahead and start a lab over here we can start a new virtual machine so let's add a new instance now this virtual machine is a black machine it only has an operating system which is linux and docker so in this terminal window if we type node look node command not found so node is not installed here but because we have docker we can pull and run the image that i published on docker hub so let me maximize this window by pressing alt and enter first let's run docker version so on this machine we're running docker version 20.10.0 so to pull and run my program first we type docker pool code with mosh slash hello dash docker alright docker downloaded this image we can verify it by typing docker what command should be run here docker images or or or image ls so on this machine we have this repository code with mosh slash hello docker and this repository contains a single image with this tag latest so now we can run this application exactly the same way we run it on my development machine so from any directory we can type docker run code with mosh slash hello dash docker and here's the message beautiful of course i had to cut this down in editing it took a little while to start this application on this very slow virtual machine but you got the point so we can take any application and dockerize it by adding a docker file to it this docker file contains instructions for packaging an application into an image once we have an image we can run it virtually anywhere on any machine with docker hey guys mosh here if you're serious about learning docker i actually have a complete docker course on my coding school codewithmosh.com this tutorial you've been watching is a cut down version of the complete course in the full course you will learn how to run and deploy a full stack application with a front-end back-end and a database our application also includes automated tests so it's a great real-world scenario for learning docker just like my other courses you will also receive a 30-day money-back guarantee and a certificate of completion you can add your resume in case you're interested you can find the link down below in the description box all right the next step in our journey is the linux command line but why linux what if you're a windows user well you still need to know a bit of linux for a number of reasons for starters docker has its foundations built on top of basic linux concepts so if you want to be productive and troubleshoot issues easily you need to know some of the basic linux commands also most tutorials online are based on linux commands so if you don't understand these basic commands you're not going to get far in my opinion learning linux is like learning english i think everybody should know some english these days you don't need to speak it or write a book in it but you need to understand it so unless you're a power linux user do not skip this section it's gonna be super easy and extremely useful so let's jump in and get started [Music] let's start up this section by talking about linux distributions also called linux distros so as you probably know linux is open source software and for this reason many individuals and communities have created their own version of linux called linux distributions each of these distributions is made to fit specialized needs like running servers desktop computers mobile phones and so on so we have ubuntu which is one of the most popular linux distributions we also have debian alpine which we briefly talked about it's a very small linux distribution we also have fedora centos and so on believe it or not there are more than a thousand linux distributions out there now most of these distributions support pretty much the same set of commands but sometimes you might discover differences along the way so be aware of that in this section we're going to use ubuntu linux because it's one of the most popular distributions but if you have a preference for another distribution that's totally fine [Music] all right let's see how we can run ubuntu on this machine from this lesson i want you to start taking notes so watch this video take some notes and then after the video repeat the steps i have shown you okay so we go to hub.docker.com and search for ubuntu over here you can see the official ubuntu image that's been downloaded more than 10 million times let's have a quick look here so for each image you can see the command to pull that image onto your machine now in this lesson i'm not going to use the pull command i'm going to show you a shortcut so here in the terminal instead of running docker pool ubuntu i'm going to run docker run ubuntu now if we have this image locally docker is going to start a container with this image otherwise it's going to pull this image behind the scene and then start a container so take a look see see see docker is unable to find this image locally and now it's pulling it from docker hub now what happened well docker started a container but because we didn't interact with this container the container stopped let me prove this to you so if you run docker ps we can see the list of running processes or running containers look we don't have any containers running here but if we type docker ps dash a for all we can see the stopped containers as well let me increase the size of this window so you can see it clearly good so we have two stopped containers the first one is using the ubuntu image this is the one that we just started and the second one is hello docker which we started earlier in the course so to start a container and interact with it we have to type docker run dash i t that is short for interactive we're going to start a container in the interactive mode and in this container we're going to load the ubuntu image which we have locally good now what we have here is called the shell a shell is a program that takes our commands and passes them to the operating system for execution okay now what we have here is called the shell prompt let me break it down for you so it doesn't look mysterious the first part root represents the currently logged in user so by default i'm logged in as the root user which has the highest privileges then after the add sign we have the name of the machine so this container has this id which is automatically generated by docker and in this case it's like the name of a machine okay and after the colon you can see forward slash that represents where we are in the file system a forward slash represents the root directory that is the highest directory in the file system we'll talk about that soon then we have a pound and this means i have the highest privileges because i've logged in as the root user if i logged in as a normal user instead of a pound we would see a dollar sign okay so in this shell we can execute a bunch of commands for example we can say echo hello and this prints hello on the terminal we can also say who am i this shows the current user so these commands that we're running here this shell program that i told you about takes these commands and passes them to the kernel for execution now let me show you something really cool if we type echo dollar sign 0 we can see the location of this shell program take a look so so so that is forward slash bin slash bash so bin is a folder or a directory and inside this directory we have a program called bash which is short for born again shell so apparently steve bourne is the first person who created a shell program bash or born again shell is a reference to steve bourne so bash is an enhanced version of the original shell program okay now one thing you probably notice is that in linux we use a forward slash to separate files and directories but in windows we use a backslash so that's one of the first differences the other difference is that linux is a case sensitive operating system so if you type echo with a capital e it's not going to work bash tells us echo command not found so lowercase and uppercase letters are different and this is not limited to commands it's applicable everywhere if you want to reference a file or a directory or a user pretty much anything we should always spell it properly with the right uppercase and lowercase letters now one last thing for this lesson using the up and down arrows we can go through the commands we have executed so far so this is a pretty useful shortcut you need to know because you don't want to type these commands manually all the time okay also using the history command we can see all the commands we have executed lately so take a look so earlier we used who am i we also used echo and so on now we can replay any of this command by typing an exclamation mark followed by the command number so if i type 2 this is exactly like running who am i now it's your turn i want you to pause the video go through your note and execute the commands i have shown you in this video [Music] these days most operating systems and development platforms come with a package manager you've probably worked with tools like npm yarn pip and so on here in ubuntu we also have a package manager called apt which is short for advanced package tool so let's execute this command look this command has a bunch of sub-commands so using list we can see the list of packages we can also search for them we can show details about the package we can install reinstall and remove a package and so on now technically apt is the newer package manager we also have apt-get which you see a lot in online tutorials going forward we're going to use apt because it's easier to work with so let's say we want to install a package called nano nano is a basic text editor for linux now if you press enter here we get an error because this image this ubuntu image we are running does not have nano so this is where we use apt to install this package so if we type apt install nano we get an error saying unable to locate package nano why is this happening well here in linux we have a package database and this database might contain hundreds of packages but not all these packages are installed so if you want to see all the packages in this database we type apt list look these are all the packages now in front of these packages you can see some of them are installed but not all packages in this database are installed when we type app install nano this command looks at the package database and in this database it cannot find a package called nano so this is where we use the update command to update the package database now let me press enter then x is going to all these sources like security.ubuntu.com and all these other websites to download the list of packages so now our package database is updated so if we run apt list we see far more packages and as you can see most of these packages are not installed because we don't have installed in front of them so now we can run apt install nano and and and nano is installed so here's what you need to take away before installing a package you should always run apt update to update your package database and then you can install a package like nano now we'll talk about nano later in this section but before going forward let's make sure that this package is installed properly so if you type nano great so here we have a text editor we can type something let me resize the window so down below you can see the shortcuts to exit we have to press ctrl and x now it's asking if you want to save the changes no so we're going to press n good we're back here we can clear the terminal window by pressing control and l okay so we have installed nano now let's say we want to remove it so we type apt remove nano it's asking for confirmation let's go ahead great so nano is gone if i type nano we get this error saying no such file or directory now here's a little exercise for you in this image we don't have python so we get an arrow so i want you to use apt to install python in this image make sure it works and then remove it [Music] so in linux just like windows our files and directories are organized in a tree in a hierarchical structure so in windows we have a structure like this with c drive on top of the hierarchy then below that we have directories like program files windows and so on in linux we have the root directory on top of the hierarchy below that we have a bunch of standard directories for example we have bin which includes binaries or programs we have boot which includes all the files related to booting then we have dev the first time i saw this i thought this is short for development it's not it's short for devices so in linux everything is a file including devices directories network sockets pipes and so on so the files that are needed to access devices are stored in this directory then we have etsy there are different opinions what is this short for but one common opinion is this is short for editable text configuration so this is where we have configuration files we also have home this is where home directories for users are stored so on a machine with multiple users each user is going to have a home directory here we also have root which is the home directory of the root user only the root user can access this directory then we have lib which is used for keeping library files like software library dependencies we have var which is short for variable and this is where we have files that are updated frequently like log files application data and so on and finally we have proc which includes files that represent running processes so once again in linux everything is a file processes devices even directories are files now you don't need to memorize any of these directories i just listed them here so as we go through the course these directories look familiar to you that's all next i'm going to show you how to navigate the file system and there you will see these directories one more time [Music] let's see how we can navigate the linux file system so the first command we're going to talk about is pwd which is short for print working directory with this we can see where we are in the file system so a forward slash represents the root directory now to see the files and directories here we type ls which is short for list so we have bin which we talked about earlier that is short for binaries this is where we have binary files and programs we have dev which includes files for devices we have boot which includes boot files etsy which includes configuration files and so on now by default ls lists these items on multiple lines if you don't like this layout and want to show one item per line you need to pass an option that is dash one you might prefer this layout we have another option dash l for seeing a long listing this listing includes more details so in the first column we have the permissions of this file or directory the first time you see this it might look really scary but trust me it's easier than you think we'll talk about that later in this section over here you can see the user that owns this file or directory we can see the size we can see the date and so on now to change the current directory we use the cd command we have the same command in windows now here we can type a relative or an absolute path a relative path is relative to where we are so in this root directory we have directories like bin boot and so on so this is a relative path now in contrast an absolute path always starts from the root directory so let's go to a directory starting with e now we can press tab to get auto completion beautiful now let's go to a directory starting with a now if you press tab nothing happens because we have multiple directories starting with a so we have to press tab one more time great so we have three entries three directories add user.conf alternatives and apt so let's type the second letter p and then press tab beautiful now let's see what we have in this directory so these blue items are directories and sources.less is a file now to get out of this directory we can go one level up by typing two periods again i'm pretty sure you're familiar with this but i want to cover it to make this section comprehensive so we can go one level up to get to the etsy directory or two levels up to get to the root directory look we're currently in the root directory now when using ls we can optionally specify a path let's say i'm somewhere in the file system but i want to look at the content of another directory i don't want to navigate to that directory so i'm currently in the root directory i want to know what files and directories we have inside the bin directory so once again we can type a relative path or an absolute path starting from the root directory so here are the binaries in this directory look at pwd that's the command that we just executed so pwd is a program in this directory here's another example look at echo so most of the commands we have been exploring so far are programs or binaries in the bin directory okay now let me show you a shortcut earlier i told you that here we have this home directory where each user has a home directory but the root user has a special home directory called root now to get here there are two options we can type an absolute or relative path and go to root but there is a shortcut so let me get out of this directory so i'm currently in the root directory to get to my home directory i can type a tilde and this applies to all users not just the root user whenever we type cd tilde we go to our home directory now right now there is nothing here but in the next lesson i will show you how to create some files in this directory so before moving on to the next lesson i want you to spend a couple of minutes and play with the commands we explored in this lesson navigate the file system get adventurous see what you can discover i'll see you in the next lesson [Music] let's see how we can manipulate files and directories so i'm currently in the root directory and i want to go to my home directory how can i do that do you remember we type cd tilde great now in this directory i want to create a directory called test so we type mkdir test let's verify it's right here as i told you before blue represents a directory now let's say we want to rename this directory how do we do that we use the move command with this we can rename files and folders or move them somewhere else so we can move or rename test to docker all right beautiful now let's go in this directory to create a new file we use the touch command so we can create hello.txt look we have this new file here this is a new empty file in the next lesson i will show you how to edit files so for now don't worry about it also using the touch command we can create multiple files in one go so we can create file 1 file 2 and file 3.

now take a look beautiful now i've got a question for you how do we list these files with a single file per line do you remember we type ls dash 1. now let's say we want to rename hello.txt to something else once again we can use the move command so we type h press tab to get auto completion we can rename this to hello docker.txt or or or we can move it to a different directory for example we can move it to the etsy directory and here i'm using an absolute path because my path has started from the root directory now in this lesson i don't want to move this anywhere so i just want to rename this to hello docker.txt and if you're wondering how i remove this entire word in one go i press ctrl and w so let's bring it back hello docker.txt take a look good now let's see how we can remove one or more files to do that use the rm command like here we can type one or more files so we can say file1.txt file2.txt or we can use a pattern for example i can say i want to remove all files that start with file let's verify it beautiful we only have hello docker now let's get out of this directory and remove the directory itself so we type rm rm rm docker now we get an error saying docker is a directory so to remove directories we have to use the r option which is short for recursive so we want to remove this directory and all its content recursively now we press d tab beautiful so let's verify there is nothing here so let's quickly recap we use mkdir to create a new directory we use touch to create a new file we use move to move or rename files or directories and we use rm to remove files and directories now as an exercise go to your home directory create a bunch of files and directories rename them and then remove them in the next lesson i'm going to show you how to view and edit files [Music] all right let's see how we can edit and view files so earlier we briefly talked about nano now it's a basic text editor for linux now on this image that we are running we don't have nano so we have to install it do you remember the command for installing nano that is apt install nano all right now we have nano so we can launch it and optionally supply a file name let's say file1.txt so here we have a basic text editor we can type whatever we want and when we're done look down below the shortcut for exiting is control and x so i'm going to press that now it's asking if you want to save the changes so we press yes and here we confirm the file name we can keep the same file or change the file name let's go ahead so now in this directory we have file on the txt beautiful now to see the content of this file we have a few different commands the simplest one is cat and this has nothing to do with cats it's short for concatenate so later i will show you how we can use this command to concatenate or combine multiple files but with cat we can also see the content of a file so if we say cat file1.txt we can see the content of this file now cat is useful if our file is short and fits on one page but if you're dealing with a long file it's better to use the more command let me give you an example so so so i'm going to use cat to show the content of this file slash etsy slash add user.conf this is a really long file so if i scroll up look we have a lot of text now sometimes you don't want to see all the content in one go you want to scroll down you want to go page by page this is where we use the more command so more etsy slash adduser.conf now down below look you can see more 15 percent so we're seeing 15 percent of this file now if we press space we can go to the next page so now we're at the position 33 alternatively we can press enter to go one line at a time now the problem with more is that we can only scroll down we cannot scroll up so to do that we have to use a different command called less so to exit here we press q good now in this image we don't have less so once again we have to manually install it so apt install less so less is a newer command that is supposed to replace more let's go ahead great so let's look at the same file using the less command hcl slash adduser.com so now using the up and down arrows we can scroll down or up so just by pressing up and down arrows we also have space we can go to the next page and enter just like the more command and when we are done we can press q now we have a couple more commands for viewing the content of a file we have head which we can use to display the first few lines so here we can supply an option and say the number of lines we want to see is 5. let's look at the same file so this shows the first five lines of this file similarly we have tail which shows the last few lines so let's apply an option and say we want to look at the last five files that's it so to recap we can use nano to write something to a file we can use cat to view the content of small files less to view the content of long files in an interactive way and head and tail to view the first view or the last few lines of a file [Music] one of the important concepts in linux is the concept of standard input and output so standard input represents the keyboard and standard output represents the screen but we can always change the source of the input or the output this is called redirection let me show you using a few examples so we talked about the cat command to see the content of a file let's say file1.txt when we execute this command this command or this program reads data from this file and prints it on the standard output which is the screen that is why we see the content here but using the redirection operator which is the greater than sign we can redirect the output from the screen to let's say a different file so now cat will read the content from this file and write it to this file take a look so we have file to.txt if we view it we see the exact same content as file one okay now what is the point of this well earlier i told you that we can use the cat command to concatenate or combine multiple files so here we can say cat file1.txt and file2.txt if you press enter cat is going to read the data from both these files and print it on the terminal which is the standard output but once again using the redirection operator we can write the result to a different file combined the txt so this is how we can use this command to combine multiple files now the redirectional operator is not limited to the cad command we can use it pretty much anywhere for example earlier we talked about the echo command if we say echo hello we see the result on the terminal but if we say echo hello to hello.txt now we have a new file here hello.txt which contains hello so if you want to write a single line to a file we don't have to use nano we can use the echo command so echo whatever to whatever file the txt okay now here's a little exercise for you i want you to get a long listing of the files in the etsy directory and write the output to a file so pause the video and work on this for a few seconds so here's the solution to get a long listing we type ls-l then we specify the path the exit directory now instead of printing the result on the terminal we're going to write it to a file called files.txt okay now let's view this file perfect now one more thing before we finish this lesson using the greater than sign we can redirect the standard output but we also have the less than sign to redirect the standard input i personally haven't found many use cases for this so i didn't cover it in this lesson all right that's all about redirection next we're going to talk about searching for text and files hey guys mosh here if you're serious about learning docker i actually have a complete docker course on my coding school codewithmosh.com this tutorial you've been watching is a cut down version of the complete course in the full course you will learn how to run and deploy a full stack application with a front-end back-end and a database our application also includes automated tests so it's a great real-world scenario for learning docker just like my other courses you will also receive a 30 day money back guarantee and a certificate of completion you can add your resume in case you are interested you can find the link down below in the description box [Music]

docker a tool that can package software into containers that run reliably in any environment but what is a container and why do you need one let's imagine you built an app with cobalt that runs on some weird flavor of linux you want to share this app with your friend but he has an entirely different system so the problem becomes how do we replicate the environment our software needs on any machine one way to package an app is with a virtual machine where the hardware is simulated then installed with the required os and dependencies this allows us to run multiple apps on the same infrastructure however because each vm is running its own operating system they tend to be bulky and slow now a docker container is conceptually very similar to a vm with one key difference instead of virtualizing hardware containers only virtualize the os or in other words all apps or containers are run by a single kernel and this makes almost everything faster and more efficient there are three fundamental elements in the universe of docker the docker file the image and the container the docker file is like dna it's just code that tells docker how to build an image which itself is a snapshot of your software along with all of its dependencies down to the operating system level the image is immutable and it can be used to spin up multiple containers which is your actual software running in the real world create a docker file and use from to start from an existing template like ubuntu this base image gets pulled down from the cloud and you can also upload your own images to a variety of different docker registries from there you might want to use run to run a terminal command that installs dependencies into your image you can set environment variables and do all kinds of other stuff then the last thing you'll do is set a default command that's executed when you start up a container and now we can create the image file by running the docker build command it goes through each step in our docker file to build the image layer by layer we can then bring this image to life as a container with the docker run command as your app demands more resources you can run it on multiple machines multiple clouds on-prem or wherever you want reliably this has been docker in 100 seconds if you enjoyed it make sure to like and subscribe and stay tuned for more docker content coming to this channel soon thanks for watching and i will see you in the next one

today let's learn about Docker this tutorial is for anyone wanting to learn how to use Docker in a hobby or professional setting we will start from the absolute Basics and include everything you need to get started including what Docker is and what problems it solves how Docker containers differ from virtual machines how to install Docker which you can do on Windows Mac and Linux what's the difference between containers and images pulling running and debugging containers Port mapping Environmental variables and arguments persistence using volumes and bind mounts building your own Docker images optimizing images and multi-stage builds using Docker compos to orchestrate multiple containers where we'll put it all together to make a fully functioning application using an enginex front end serving HTML a python backend using fast API with a mongodb database as well as how to publish your Docker images to an image registry like dockerhub and and some final notes about deploying containers in the cloud this video is sponsored by Docker that's right the company that makes the tool is also sponsoring this video which is awesome because Docker is a tool that I actually use pretty much every single day in fact I've been using Docker for longer than this channel has even existed so it's great to have their support let's get to it what exactly is Docker and why would we want to use it I think it's easier to start with the problems that Docker solves have you ever got some code working on your machine than transferred it to another computer or given it to a coworker and it doesn't work anymore or Worse the tests pass on your machine but fail in production have you ever burnt an entire day trying to install a program and no matter how closely you follow the docs it just doesn't seem to work have you ever installed an application only to have a seemingly unrelated system upgrade break your application have you ever had two applications both individually work fine but when you try to install both of them they don't work anymore these kinds of problems can range from being an annoyance all the way up to crashing your app in production the underlying cause of all these issues is that your application depends on more than just your source code you need to make sure that the right versions of all your dependencies are installed that your operating system has all the necessary packages installed that any Services config files users and permissions are set up correctly and that none of these hidden dependencies conflict with the hidden dependencies of any other part of your application or the rest of the system that's where Docker comes in Docker is software for deploying and running containerized applications Docker solves all these problems using containers which are isolated groups of processes each container has its own file system that has everything it needs to run like your app dependencies system packages and so on so all the issues with conflicting hidden dependencies or system upgrades on the host breaking your code just disappear containers are also isolated from each other and the host using kernel isolation features so containers have a limited ability to affect each other Beyond competing for system resources or competing with each other over the network unless you specifically give them extra permissions you may be wondering this sounds a lot like using a virtual machine and you're right using a virtual machine with a custom image could provide a lot of the same benefits however if you've ever tried running a virtual machine before you probably noticed that virtual machines tend to be significantly slower than the host this is because virtual machines emulate well entire machines the kernel Hardware drivers Network stack everything and that's a ton of overhead just to run a single application in contrast Docker images are made up of normal processes running on the host system isolation makes them feel like their own little machines but they actually use your existing kernel your existing Hardware your existing Network so you get much closer to Native performance and have much more efficient use of system resources than with a virtual machine so while virtual machine images are typically gigabytes big and take minutes to start Docker images are usually only megabytes and start in under a second next let's install Docker if this is your first time using Docker you probably want to use Docker desktop which is available for Windows Mac and Linux and comes with everything you need to use Docker including both command line and guey interfaces if you are more experienced and prefer the command line only then I'd recommend installing Docker engine which comes with the core functionality but no gooey Docker desktop does more than just provide a gooey though Docker was originally developed for Linux and most Docker containers are for Linux so Docker desktop uses a hypervisor layer to run a lightweight Linux drro which is what allows you to be able to run Linux containers even when you're on Windows or Mac I'll show you what I do to install Docker but keep in mind that setup instructions change over time so always refer to docker's official installation instructions for how to install for your system in particular make sure you're running an operating system version that is specifically listed as supported I'm on Linux using a fresh install of Ubuntu 20204 as always make sure your system is up to date before installing anything next I'll set up Docker package repository following the instructions at this link all you need to do is copy and paste this code into your terminal then I'll download and finally AP install the Deb file it looks like we got an error but according to the docs this is expected and we can just ignore it and that's that's it let's start up Docker desktop in order to start the docker demon which is what manages containers under the hood review and accept the service agreement and I'll go ahead and continue without signing in for now skip the survey and we're in you can set it to run on Startup in the settings if you want we can now run Docker commands so let's start by running hello world we did that with Docker run hello world and as you can see it pulled the latest hello world image and ran it if we run hello world again it doesn't download the image again since we already have it downloaded if you wanted to download the latest image or Polo a specific image without running it you can use Docker pole in this case it just confirms that indeed we have the latest image that we downloaded just a few seconds ago and a quick security note by default Docker runs with root privileges because it needs to be able to do things like bind to privileged ports Mount file systems and modify network settings it's possible to run Docker without rout but that's beyond the scope of this video now that we can run Docker commands we should get some of the basic concepts down the first one is the difference between Docker containers versus images take a look at our containers tab where we have two separate containers that both have an image of hello world whereas in our images tab we only have one image hello world the relationship is that images are templates for containers images specify the contents of the file system what applications are installed users environmental variables and the default command that will run when you start a container you can think of an image as a recipe for running an application containers on the other hand are the actual groups of processes that are launched that follow the instructions the image specified every time I use Docker to run an image it creates a new container based off that image that means I can have many containers all based off the same image even running at the same time you can see your images and containers in their respective tabs or Docker image LS will show us images and Docker PS will show us running containers our hello world's already exited so we see nothing if we want to see them then we use A- a flag to include all containers including stopped ones let's try running something with a little more substance than hello world how about an engine X web server engine X is publicly available on dockerhub which is the default image registry the docker uses if you don't specify so we can use Docker pull engine X or in our case Docker run engine X and that downloads and runs the latest version in this case 1.27 0 it looks like it downloaded and started up fine this output here is the logs for the container we can see it's still running in the guy but when we visit Local Host we don't see anything that's because by default containers are isolated from the host Network in order to interact with the engine X server we need to publish the port it's listening on so our host can see it we do that using the- p flag let's stop it and try again now when we go to Local Host we see the default engine X page in this case we wanted to publish the containers Port 80 to our own port 80 but you can remap ports however you like if we wanted the engine X page to be available on Port 5000 we could run the same command with 5000 instead of 80 now the containers Port 80 is available on our Port 5000 I've also been running this container in the foreground so far and I've been stopping it using control C if you want to run the container in the background you can slap on A-D for detach now my terminal stays free this string that it printed out is the container ID which I can use to interact with the container while it's running or to help me locate it in the guey we can see part of that ID here with Docker PS for most commands that need a reference to a container you can refer to it either by the container ID or by its name you can specify a name when you launch the container using D- name otherwise one will be randomly generated that's what we see here a randomly generated name if I wanted to see the logs that were being printed out before I could use Docker logs and then paste in either the name or the container ID or of course I can go to Docker desktop find the container in the list open it up and look at the logs in the gooey you can also stop containers in the guey as well or by using Docker stop we've also started to accumulate a bunch of containers here these will not be deleted unless you delete them which can be done in the guey or with Docker container prune this will remove all stopped containers or you can tell Docker to automatically delete a container when it stops by using the-- RM flag in the Run command next let's talk about tags you may have noticed that in our Docker run command we just said engine X but what version of engine X are we running here if you don't specify you just get the latest version or the latest version you already have downloaded we can pick a specific version by adding a colon and a tag engine X 127 will give me the enginex image that's tagged with the label 1.27 but be warned Docker tags are like symbolic links engine X 127 points to some image currently it's an image running debb and Bookworm with engine X 1.27.0 but when engin x127 do1 gets released they'll most likely update the 127 tag to point to a new image that's running engin x127 do1 to mitigate that risk you can use a more specific tag like engine x1270 D Bookworm that's very unlikely to be updated to a new image or you can pin the image using its digest which uses the shaw 256 hash function the digest was printed out the initial time we pulled the image or we can find it using Docker image ls-- digests or it's also displayed on dockerhub you can go to the tags list and then search for the tag that you're interested in look for it in the list here in this case we want 127 Bookworm and here's that same digest then whenever you specify the image put an at with the digest at the end if you provide a digest you can leave the tag there for humans but it's ignored pinning by digest is the only way to have high confidence that no changes are introduced in your containers environments unless you specifically asked for them for a production application I highly recommend pinning by digest not by verion this is effectively the same advice that I give for git tags and for python package versions tags are great for humans but they don't provide the kind of stability that hashes do but if you're just playing around or using private images then tags are fine moving on let's talk about environmental variables and arguments just like normal programs containers can be run with environmental variables and arguments passed at runtime just to demonstrate let's run Python and pass it some environmental variables you pass environmental variables with- e flags and you pass the command and args to run just after the image here we're still going to run the default python command and we pass it the- C that allows us to give an inline script to run that prints out the environment the output shows are abc123 and rdf 456 but uh I don't know if you noticed because the weight time was cut out in the video but this python image is over a gigabyte big time to talk about slim and Alpine images in production and testing settings image size Matters a lot python like many other popular images provides tags that come with various different environments with various different things pre-installed the default python image is currently based on a fully feature Debian install in contrast any of the tags that have Slim in them are based off a much more barebones version of Debian pulling that image we see it's nearly 10 times smaller than the non-sim version if that's still too big keep an eye out for tags that have Alpine in them Alpine refers to Alpine Linux which is a lightweight Linux drro focused on minimalism pulling the Alpine image we see it comes in at only 55 mbes around half the size of the slim version and 20 times smaller than the main version I recommend using either the slim or Alpine images if they're available however while slim is usually an easy win over the non-sim version choosing between slim and Alpine is not as obvious Alpine Linux is significantly different than Debian Linux Alpine uses muscle libc instead of GBC it doesn't come with bash and it uses the APK package manager instead of Apt so there's a learning curve and also a risk of suddenly different Behavior than you may be used to on a Debian based distro next up debugging containers let's say we have a container running and something's not quite working the way we want to yes we can inspect the logs but sometimes you just need a shell to poke around the two main ways of doing this are number one using the goey clicking the this exec tab gives you a root terminal where you can play around alternatively you can use Docker exec at the command line which allows you to execute a process on an already running container the dashit stand for interactive and TTY interactive keeps standard in open allowing you to interact with the process and TTY allows terminal features like command history and autocomplete to work properly the specifics aren't that important just remember if you want an interactive session like a shell or a repple with Docker run or Docker exec use-it of course now we're root inside the container so we can debug to our hearts content moving on we need to talk about persistent storage from everything we've seen so far every time we run a Docker container we start fresh with a pristine container generated from the same image that starts with the same file system every time any files written from the previous runs are gone we can demonstrate this with a simple python script all it does is open a file right to it and then print out the contents of the file but every time we run it we still just see one copy of ran because it's starting clean every time but many applications like databases have data that they want to persist across runs even to persist across image changes like upgrading the database version we can achieve that using a mount you can add a mount to your container using- V which stands for volume or using the more veros d-mount syntax this time the SL dat directory is populated by a volume that lives beyond the life cycle of the container so each time we run the script we see the previously saved data currently in Docker there are three kinds of mounts volumes bind mounts and Tempest mounts volumes and bind mounts are used for persistence whereas Tempest mounts aren't persisted and are beyond the scope of this video volumes are newer more fully featured and are completely managed by the docker demon whereas bind mounts mount a local file or directory from the host into the Container the syntax of using a volume versus bind Mount is nearly identical they both use the- V flag it just depends on whether you give it a name like my data or a relative or absolute path like slashy data or/ my data if you give it a path then Docker will Mount The Source path into the target path as a bind Mount if you give it a name then Docker will mount a named volume into the target path let's try creating a my data directory and passing my data to the- V it seems like this version also works and persists our data the only apparent difference right now is that we can directly see and manipulate the contents of the my data directory whereas the my data volume is stored in some location managed by Docker that we don't really know about so which one should you use it really depends on why you want to share the data with the container in the first place when you're developing code on your local machine bind mounts tend to be what you want because their main benefit is that they're simply mapping a file or directory on the host into the Container you can see all the files right there inspect them in your IDE or with other programs and change them as you please this can be really useful to quickly modify an image to your liking by sharing a config file or build artifacts between the container and the host it allows you to quickly iterate without going through the process of building your own image and rebuilding it every time you make a change watch out though mounting a host directory means you're giving the container direct access to that directory meaning a container can modify or delete files or change permissions of files on the host this risk can be mitigated by setting them out to be readon which is what this colon r at the end does but sometimes containers just need right access a common annoyance is when a container CH owns the directory to root within the container because it's a sensitive config file which then makes your current user unable to view or modify the f files until you pseudo CH own them back volumes on the other hand tend to be much more appropriate in a production setting because of the additional features they provide that are useful in Cloud environments since volumes are managed by the docker demon and not the host file system volumes can be easily shared across multiple containers even containers running on different hosts or with the underlying data stored on a remote host or cloud provider such as in an Amazon elastic file system an NFS share or an SSA P file system volumes also abstract away the file system of the host which removes the risks associated with giving a container access to files on the host but the flip side of that is that you can't conveniently view or modify the contents of a volume from the host I mean if you use a local volume the files are all there it's just not convenient the primary way you're meant to interact with a volume is through containers not directly from the host but for many use cases you might not care to actually see those files unlike a config file that I might want to go back to reference and iterate on the underlying storage for a database is typically not in a human readable format anyway so providing a directory in my project folder for a database to store all its junk is more of a nuisance than anything in that case a volume would be more convenient than a bind Mount even in a development setting so use volumes use bind mounts don't let anyone tell you you're doing it wrong especially when you're learning just do what works for you so far we've seen how to use existing Docker IM images but we only customized them through environmental variables arguments and mounting files and directories if you ever want to customize beyond that Docker makes it super simple to build your own images I went ahead and installed all my Dev tools let's open up a new project folder in vs code I always recommend creating a project folder when you're creating Docker images to stay organized let's start by creating an image with an engine X server that serves custom HTML instead of the default that that's a front-end service so let's make a front-end folder and a single file named Docker file make sure to install the official Docker extension to get autocomplete and syntax highlighting although Docker files completely specify the file system we don't have to start from scratch you can and usually do use another image as a starting point by using the from keyword let's base our image on engine x170 at this point we can already build our image using the docker build command take note that the- t flag specifies the name and optionally a tag and don't leave off that dot at the end the dot specifies the build context which tells Docker where it's allowed to look for files in this case the current directory by default Docker uses the docker file at the root of the build context or you can specify the path to a Docker file with- F anyway we see that Docker created our image and now it's listed in our Docker desktop we could run it too but right now it's just a copy of engine X which we can see by comparing the hashes so let's customize it let's create a static folder for our HTML and put a minimal index.html that maintains a to-do list feel free to pause to check out what the JavaScript is doing it's pretty simple we just have a single function to add an item to the list that also creates a button to delete that item starting from an engine X image all we want to do is copy this code from the static file into the directory that enginex expects its HTML we do exactly that with the copy command the sources of a copy command are relative to the build context in this case the front-end directory whereas the destination of the copy is a path inside the images file system let's go ahead and build and run our image and voila we see the most beautiful website known to man if we exact into the container and navigate to/ user share engine X HTML there we see the contents of our static folder the engine X image left this 500 error page there so let's delete that using a run command before copying everything over run just runs a command inside the image then let's build and run again and there we go we have a nice and clean directory with just our index file I want to point out how fast the build process was how is it that we're specifying an entire file system but it takes basically no time to build the answer is due to a fundamental Concept in Docker layers layers are like image diffs at build time the docker file is run in order and each command that you run in your Docker file creates a new layer based off the previous one because layers act like diffs a small change like deleting a directory or copying a folder results in a small diff layers are also immutable so even an operation like a delete creates a new layer it doesn't modify an existing one importantly Docker doesn't just store the final image it stores all the individual layers which makes it possible to reuse existing layers since we already downloaded the engine X image Docker already has all its layers cached for immediate use and just needs to apply two tiny diffs to create our final image that's what we're seeing here Docker desktop lets you inspect the layers in an image and the commands that created them which is a great way to learn about Docker files at build time Docker attempts to use cached layers wherever possible Docker will check the current command against the last time you built and reuse the result unless the cache was invalidated different commands have different caching rules but the main ones to keep in mind are number one changing a command invalidates the cache for the current and all later commands so if I change this 127.0 to 127.0 it will have to download all the new layers from 1271 that were in 1270 then it will have to rerun the run and copy regardless of whether anything else changed but if I changed the Run command it would only rerun the run and copy if I changed the copy it would only rerun the copy and rule number two most commands are cached by default even if they aren't pure functions this run command is pure the resulting image relative to the previous one should be the same every time regardless of when you run it but in general a run command can do almost anything you could apt get install some package today and tomorrow the same command might download a different binary but Docker will cach and reuse the result from today if you want you can force Docker to not use the cache with-- noach in your build command but don't worry the copy command additionally uses file metadata to invalidate the cache so if your Source contents change it will at least invalidate copy commands even if your Docker file doesn't change we'll see some caching tricks to speed up builds later on and one final security warning about layers before we move on to reiterate a run command that deletes something does not modify the previous layer it creates a new layer on top of it that means it's not safe to put any sensitive information inside a Docker image even if you delete that information in a later command layers are like your git history creating a new commit with a certain file deleted will not stop a malicious user from pulling that data out from the history now that we have a front end working let's create a simple python backend to store or to-do list so that it can persist across refreshes go ahead and create a backend directory with source and mysite directories within it with a main.py file also remember to create and activate a new virtual environment go ahead and pip install fast API and then use pip freeze to create your requirements txt file again in a production setting I recommend pinning notches versions but hashes as well but that's out of the scope of this video so we'll just pin the versions in our main.py let's start with some imports and creating the fast API object next we create two pantic models for our to-do items an item will have an ID and some content but creating an item will just require the user to give us the content we'll store the to-dos as a global in memory list and just use a global counter to generate IDs then we'll have three end points 1 2 to create to-do items one to list all the to-do items and one to delete a to-do item by its ID feel free to pause here if you'd like to check out the implementation of the endpoints but they should be pretty straightforward next let's create a Docker file at the root of our backend directory we'll base our image on python 312 slim this worker command sets the working directory inside the image it's common to store your python project in an app directory like this then let's copy over our requirements txt and run pip install to install our requirements after that we copy over the rest of our source we could have copied the requirements and Source in a single copy command but splitting them is actually intentional remember if nothing changed then Docker will cach previous commands including this run requirements files change far less frequently than source files so I copy and install the requirements first then as long as no requirements change a change in just a source file will not require rerunning the PPP and stall command which is the slowest part of the build we also give pip the no cter option because the cter would be inside the image which would be pointless next we use the expose command which I think is quite confusing this is not a way to automatically expose a port to the host whether you expose in your darker file or not you will still have to use the- P flag when you run it in order to publish the port to the host in older versions of Docker expose actually did something meaningful but the modern purpose is that the the expose command is basically just documentation for the reader but I still use it because documentation is good finally we have the command command which specifies the default command that will be used if none is supplied when you run the container this command will be executed using /bin/sh but there's also an entry point command if you want to specify a different executable command and entry point also allow you to specify either a string or a list of strings like we do here I always recommend using the list of strings approach great let's build it and run it remembering to publish Port 8000 looks like it started up just fine let's open it in the browser open up slash docs and we can see our fast API Swagger docs with our three end points next I've updated the front end to actually call the back end I've just hardcoded the URL here and I'm using the built-in JavaScript fetch we list our to-dos using the get endpoint we add an item using the post endpoint and we delete an item using the delete endpoint let's rebuild and run them both Let's test it out and as per always we forgot to add the co middleware so the browser is blocking our requests cores is beyond the scope of this video but the short answer is that we access the website from Local Host on Port 80 but the JavaScript is calling out to Local Host on Port 8000 for security purposes those are treated as two completely different websites so the browser treats that as malicious behavior and blocks it unless the response specifically indicates that it's okay back in our main file we just need to import and add this course middleware let's rebuild and run it again and now our backend and front end are both up and running and talking to each other just to show you that it's working let's add some items to the list and see that even after a refresh we still have our items in the list because that's now persisted by the back end well as long as we don't shut the back end down at this point I want to show a neat trick with Docker files that helps both with security and can drastically lower the overall image size I'm talking about multistage builds take a look at this Docker file again the problem here is that the final image includes the things that were only needed during the build process and are not actually needed in production our example is pretty minimal so there's basically just this requirements.txt that isn't needed at runtime but in a General application your build dependencies and runtime dependencies can deviate significantly they can include things like heavy build systems a c compiler and so on and the build process itself can generate many unwanted files like caches and logs this can also be a security risk because having build tools available on a production machine leaves a much greater attack surface introducing multi-stage builds what we can do is separate the commands used for building versus running the application into separate stages in our build stage the goal is to produce all the wheels we need to install so we can just copy them into our run stage then all the Run stage needs to do is install those pre-built wheels and run in order to make this work we'll need to make our project and installable package first which I recommend doing anyway so let's add a p project. tml file to make our code installable here's a pretty minimal one using setup tools we just set our project name python version our project version and then any hard dependencies and then we specify where to find our packages I also added this empty init.py that's all we needed to do in order to make our package installable so let's get back to the docker file this time we copy over our infrequently changing project files but instead of pip installing into our environment we use pip wheel to create pre-built installable Wheels which are like glorified zip files then we copy over our source and build a wheel for that as well we Define more stages by using more of these from commands and since we have more than one we should give them both names in the runner we Ed this copy command with this special-- from to tell Docker The Source location is not from our build context but rather from the previous Builder stage we copy all the wheels over install them then delete the wheels I snuck in another optimizing trick here notice how we're using a single run command with two commands in it by grouping commands together we can decrease the number of layers and potentially squash creates and deletes that cancel out although that doesn't apply in this case since our Wheels were not generated in this run command now no matter how complex the build process was the runner image has no build dependencies we started fresh and directly installed the production Wheels time to rebuild in this case Docker automatically selected the runner as our final Target but you can have many stages in a multi-stage build like a builder Dev test and prod Runner but if you have a specific one you're looking to build then you can use D- Target and specify the stage name you could also use multiple Docker files instead just do what works for you as you may have noticed by me fumbling around it can get pretty cumbersome to manually launch everything when you have multiple containers they may contain different ports they need to publish have environmental variables volumes and so on that's because Docker files are for individual components like our front end our backend but our application isn't just a single component our application is a composition of components all orchestrated in a particular way we have a front end back end maybe a database a load balancer a reddis cache a task q and so on for these compositions of components we use a higher level tool called Docker compose create a Docker composed. yl file at the project rout we specify each of our components as a separate key key under the top level Services key for the back end let's give the image and container a name and since we're building the image we need to add build info we want Docker to build the back end as if the backend directory was the root in the build process so we set the context to be the backend folder by default it'll use the docker file named Docker file but I always like to specify it by name since you might have more than one the back end is a multi-stage build so I also like to be explicit about the stage we want as the final result so we set the target to be the Runner stage then we can specify the ports that we want to publish just like we're using the- p flag in Docker run unlike the expose command and the docker file this ports configuration will actually map ports between container and host add another top level key for the front end which is similar we have an image and container name and we specify a build the front end is not multi-stage so we don't need to specify a Target and we expose Port 80 in order to avoid accidentally pulling some random person's image from Docker Hub that might have the same name you can either set the pull policy to never or you can change the image to point to a repository you own for now let's just set the pull policy to never next let's use Docker compos to build all the images now that all the images are built we can start everything all at once using one command Docker compose up if you've never seen this before it's pretty cool one command and her whole stack is up and running you can see the logs from both of the containers here you can use the dashd flag just like you can with Docker run in order to do everything in the background we can also see everything nicely grouped together in the docker desktop goey you can shut everything down by clicking this button or with contrl C at the terminal or with Docker compose down Docker compose down actually deletes the containers so if you just wanted to stop them you could have used Docker compose stop and Docker compose start to start them back up again though I usually just use up and down I hope you can see this is way more convenient than starting and stopping everything manually and using compose also creates an isolated Network just for these containers as opposed to putting everything in a global default Network or having to manually manage networks yourself which I didn't even talk about letting compos do it is just easier now that we have Docker compos at our disposal let's see how to add a database to our stack so that when we shut down our server and start it back up again we don't lose all of our data in our compos file add a new service for mongodb this is another Public Image let's grab version 7012 which is the current latest release set a container name we have nothing to build and for this one we actually don't need to specify any ports to publish because we don't need to access the database from our host we only need to access the database from our backend compose creat creates a network for the services so by default all the containers in this compos file can talk to each other additionally it's set up so that they can refer to each other by their service names so when it comes time to tell our backend the connection string for the database we can just use mongodb for the host name since we need to persist our database data let's use a volume which we can specify in the compos file create a top level key for volumes and since the default volume options are fine all we need to do is specify the name of our volume we want to create let's call it mongodb data then in our mongodb service we specify that mongodb needs to mount that volume at /data DB which is where mongodb expects it lastly our mongodb needs a username and password which the image expects from environmental variables you can pass environmental variables like this but do not do this with sensitive data like passwords Docker files and composed files are meant to be tracked in your git repository therefore they should not contain any secret data instead you can pass environmental variables using an end file like this create a folder for the mongodb service and in there create a file named NV take the environmental variables and put them in this NV file then in your compose file use the end file key specifying the path to your end file immediately afterwards you should create a g ignore and add end to your get ignore so you don't accidentally commit any end files there are still security considerations around storing secrets in environmental variables at all particularly if you're running untrusted or malicious code on the same machine if you expect your server to be compromised or if you need to segregate who at your company has access to production hosts versus who has access to the secrets those kinds of concerns are beyond the scope of this video but if you're worried about that then what you need to look into is called a Secrets manager like a WS Secrets manager or Hashi Corp Vault Docker also has a built-in Secrets manager but in production this requ re Ires the use of Docker swarm mode moving on this is enough to run a database but we need our backend to actually connect to it for that we'll use the motor library which we need to add to our p project. toml and requirements files I just installed it in my local environment and then used pip freeze to update their requirements then in the code I made a few simple changes firstly we'll grab the connection string from an environmental variable create the motor client and get the to-dos collection then we update our to-do models our to-do item now uses uu IDs for the ID field and we use this underscore ID Alias because underscore ID is what mongodb uses as a primary key then we had to update the three endpoints to actually use mongodb the changes are pretty straightforward we just swap out our list operations for mongodb insert find and delete operations we also create a EnV for the back end to pass that connection string this first mongodb is the protocol then root and example is our username and password and this second mongodb is the host name of our mongodb container just to reiterate because we're using Docker compose and the mongodb service was named mongodb in the compos file that's what allows us to use the host name mongodb here and have it be directed to the other container the port 2717 is mongodb's default Port that it listens on then we go ahead and add that n to our compose file as well lastly we have one tiny change we need to make in the front end because we changed using underscore ID instead of ID add in that underscore so the deletes work properly and we should be set we're ready to rebuild and just like that with one command Docker compose up we now have a frontend backend and persistent storage in a database all in one command let's check it's all up and running on Local Host again refresh let's add some items ABC ABC ABC sleep and code then let's go back to our terminal and Docker compose down to kill everything and Docker compose up to start it all again and refresh the page it's all still there we can delete items and add new on ones just like nothing happened and the best part is a collaborator can clone this project run Docker build Docker compose up and everything will work exactly the same way that it works on my machine and pro tip during development I recommend using Express to easily explore your database just add it as another service in your compose file it exposes a web interface on Port 881 and create a Express folder with a EnV so we can pass environmental variables to it it needs the connection string to the underlying database and then a username and password for its own interface Docker compose down and Docker compose up then visit Local Host on 881 put in root and example which were from the config and now we've got a convenient way to explore our database here's our to-do list database with the to-dos collection and we can see our three to-do items sleep code and continue being able to actually view your database in a development setting makes debugging a lot easier because you can just go in see the actual data and even edit it right here back to the compos file you may wonder about the order that containers start up in in particular some containers depend on others being available for example when our backend starts up it immediately tries to connect to the database but what if the database container hasn't started yet Docker has this dependson key where you can specify for each service which other services need to be started first then Docker will make sure that the containers start in an order where nothing is started before its dependencies this can be convenient in a pinch but I don't really recommend this option I'm much more in favor of writing your code in a way that works regardless of the startup order Often by having services use timeouts retries or pinging your dependencies to see if they're ready before continuing with normal startup in this compos file none of the services actually require the others to be started first because they all internally have timeouts or retries built in but you may find it useful anyway while Docker is incredibly useful even just for local development at some point you may want to share the images that you create either to a public repository for everyone to benefit from or to a private repo probably just to use within your company this is done through what's called an image registry which is a service that hosts Docker images the default and most popular image registry is dockerhub the process for publishing is actually pretty simple make sure you have an account with the registry you're using and create a repository for your project I just created this public demos script repo under my username then tag your image log into your registry and push you can either put the image including tag in your Docker compose or you can tag the image manually using Docker tag even though I don't have to I like to specify docker.io outfront just to remind myself that pushing this image goes to dockerhub if you use another registry like one hosted by your cloud provider change that docker.io to your registry keep in mind that you can tag a single image with as many tags as you like this is how a lot of really popular images work like engine X currently engine x engin x27 engin x 1270 and engin X 1270 Bookworm all point to the same image there's no hierarchical structure to tags the engine X team literally just tag tag the same image with a bunch of different tags for your convenience you can do the same thing with your tags although personally I prefer to tag every image with a single tag that specifies its full version and never change what image a tag points to I already built this image and note that Docker compos build will tag the image with whatever you put here then we need to log in if you're on Linux there are a few Hoops you need to jump through because Docker uses the pass password manager to store your credentials so you need to set that up first first you need a gpg key which you can generate with gpg generate key then copy this and do pass init and paste that in then you can log in either from the CLI using Docker login or directly in Docker desktop then use Docker push specifying the image and tag you want to push or use-- all tags to push all tags associated with an image looks like the upload succeeded let's go back to our Repository and refresh and there it is our first ever Docker image is available to the world run it if you dare and lastly let's talk about deployment which is a whole Beast of its own you don't actually ever need to deploy your containers if you don't want to it's perfectly fine to use Docker as a local development tool only or as an easy way to install and run specific versions of specific tools you need but suppose you do actually want to deploy your containers you can download Docker onto a server you control and manually pull and run your images but that's not the typical way of doing it when it comes to deployment the conversation very quickly turns to which paid cloud service provider offers the most convenient and affordable container running product for your use case there's no one right answer here and there are hundreds to choose from so you're just going to have to do some extra research anyway that's all I've got once again huge thanks to Docker both for making docker as well as for sponsoring this video thanks to my patrons and donors and as always slap that like button an odd number of times don't forget to subscribe and comment and I'll see you in the next one

hey i'm kyle and welcome to tech squid tv we're gonna be running a ton of services from this 23 terabyte server in an upcoming video but before we get started we have to go over docker thank you to hover.com for sponsoring this episode you can get 10 off your own custom domain name at hover.com forward slash techsquid what is docker docker is mainly a software development platform and kind of a virtualization technology that makes it easy for us to develop and deploy apps inside of neatly packaged virtual containerized environments meaning apps run the same no matter where they are or what machine they are running on docker containers can be deployed to just about any machine without any compatibility issues so your software stays system agnostic making software simpler to use less work to develop and easier to maintain and deploy these containers running on your computer or server act like little microheaters each with very specific jobs each with their own operating system their own isolated cpu processes memory and network resources and because of this they can be easily added removed stopped and started again without affecting each other or the host machine containers usually run one specific task such as a mysql database or a node.js application and then they're networked together and potentially scaled a developer will usually start by accessing the docker hub an online cloud repository of docker containers and pull one containing a pre-configured environment for their specific programming language such as ruby or node.js with all the files and frameworks needed to get started home users can experience docker as well using containers for popular apps like plex media server nexcloud and many other open source apps and tools many of which we will be installing in upcoming videos docker is a form of virtualization but unlike virtual machines the resources are shared directly with the host this allows you to run many docker containers where you may only be able to run a few virtual machines you see a virtual machine has to quarantine off a set amount of resources hard drive space memory and processing power emulate hardware and then boot an entire operating system then the vm communicates with the host computer via a translator application running on the host operating system called a hypervisor docker communicates natively with the system kernel bypassing the middleman on linux machines and even windows 10 and windows server 2016 and above this means you can run any version of linux in a container and it will run natively not only that docker uses less disk space too as it is able to reuse files efficiently by using a layered file system if you have multiple docker images using the same base image for instance docker will only keep a single copy of the files needed and share them with each container alright so how do we use docker install docker on your machine and we'll provide links in the description you begin with a docker file which can be built into a docker image which can be run as a docker container okay let's break that down the docker file is a surprisingly simple text document that instructs how the docker image will be built like a blueprint you first select a base image to start with using the from keyword which you can find a container to use from the docker hub like we mentioned before ubuntu and alpine linux are popular choices from there you can run commands such as downloading installing and running your software of course we'll link the docs below once our docker file is complete we can build it using docker build followed by the t flag so we can name our image and pass our command the location of the docker file once complete you can verify your image's existence with docker images now your built image can run a container of that image or you can push it to the cloud to share with others speaking of sharing with others if you don't create your own docker image and you just want to use a pre-made one you can pull one from the docker hub using docker pull and the image name you may also include a tag if one is available which may specify a version or variant of the software if you don't specify a tag the latest version will be what's batched to run a container pull it down from the docker hub or build the image and then enter docker run followed by the image name there are of course many options available when running your container such as running it in detached mode with tacd or assigning ports for web services you can view your running containers with docker container ls and as you add more they'll appear here running a single container is fun but it's annoying to enter all of these commands to get a container running and we may want to control several containers as part of a single application such as running an app and a database together something you might want to do to run wordpress for example and we're going to be accomplishing that with docker compose in our next video where we build a 23 terabyte home server if you are going to have a secure home server you have to get yourself a domain name for ssl so you can access that server anywhere in the world thank you to hover.com for supplying you with 10 off your first domain name by visiting hover.com forward slash techsquid check out their find a domain name tool to find cool domain names with cool extensions and since we're building a home server let's just type that in and see what we get i'm going to put some domains on the screen these are currently available that are related to home server get them now while they're available and make sure to get that 10 off and support the show again by going to hover.com forward slash tech squid thank you for watching let me know if you'd be interested in a part two where we go deeper into docker and we'll go over some of the specifics of docker layers and so on and stick around coming up we're building a 23 terabyte server using docker compose if you have any questions come chat with us on discord the link to that and everything else you need is in the description below alright thank you and i'll see you in the next video [Music]

welcome to Docker 101 if your goal is to ship software in the real world one of the most powerful Concepts to understand is containerization When developing locally it solves the age-old problem of it works on my machine and when deploying in the cloud it solves the age-old problem of this architecture doesn't scale over the next few minutes we'll unlock the power inside this container by learning 101 different concepts and terms related to computer science the cloud and of course Docker I'm guessing you know what a computer is right it's a box that has three important components in side a CPU for calculating things random access memory for the applications you're using right now and a dis to store things you might use later this is bare metal hardware but in order to use it we need an operating system most importantly the OS provides a kernel that sits on top of the bare metal allowing software applications to run on it in the olden days you would go to the store and buy software to physically install it on your machine but nowadays most software is delivered via the Internet through the magic of networking when you watch a YouTube video your computer is called the client but you and billions of other users are getting that data from remote computers called servers when an app starts reaching millions of people weird things begin to happen the CPU becomes exhausted handling all the incoming requests disio slows down Network bandwidth gets maxed out and the database becomes too large too query effectively on top of that you wrote some garbage code that's causing race conditions memory leaks and unhandled errors that will eventually grind your server to a halt the big question is how do we scale our infrastructure a server can scale up in two ways vertically or horizont ially to scale vertically you take your one server and increase its RAM and CPU this can take you pretty far but eventually you hit a ceiling the other option is to scale horizontally where you distribute your code to multiple smaller servers which are often broken down into microservices that can run and scale independently but distributed systems like this aren't very practical when talking about bare metal because actual resource allocation varies One Way Engineers address this is with virtual machines using tools like hypervisor it can isolate and run multiple operating system systems on a single machine that helps but a vm's allocation of CPU and memory is still fixed and that's where Docker comes in the sponsor of today's video applications running on top of the docker engine all share the same host operating system kernel and use resources dynamically based on their needs under the hood docker's running a demon or persistent process that makes all this magic possible and gives us OS level virtualization what's awesome is that any developer can easily harness this power by simply installing Docker desktop it allows you to develop software without having to make massive changes to your local system but here's how Docker Works in three easy steps first you start with a Docker file this is like a blueprint that tells Docker how to configure the environment that runs your application the docker file is then used to build an image which contains an OS your dependencies and your code like a template for running your application and we can upload this image to the cloud to places like Docker Hub and share it with the world but an image by itself doesn't do anything you need to run it as a container which itself is an isolated package running your code that in theory could scale infinitely in the cloud containers are stateless which means when they shut down all the data inside them is lost but that makes them portable and they can run on every major Cloud platform without vendor lock in pretty cool but the best way to learn Docker is to actually run a container let's do that right now by creating a Docker file a Docker file contains a collection of instructions which by convention are in all caps from is usually the first instruction you'll see which points to a base image to get started this will often be a Linux drro and may be followed by a colon which is an optional image tag and in this case specifies the version of the OS next we have the working directory instruction which creates a source directory and CDs into it and that's where we'll put our source code all commands from here on out will be executed from this working directory next we can use the Run instruction to use a Linux package manager to install our dependencies run lets you run any command just like you would from the command line currently we're running as the root user but for better security we could also create a non-root user with the user instruction now we can use copy to copy the code on our local machine over to the image you're halfway there let's take a brief [Music] [Applause] intermission now to run this code we have an API key which we can set as an environment variable with the EnV instruction we're building a web server that people can connect to which requires a port for external traffic use the expose instruction to make that Port accessible finally that brings us to the command instruction which is the command you want to run when starting up a container in this case it will run our web server there can only be one command per container although you might also add an entry point allowing you to pass arguments to the command when you run it that's everything we need for the docker file but as an Added Touch we could also use label to add some extra metadata or we could run a health check to make sure it's running properly or if the container needs to store data that's going to be used later or be used by multiple containers we could mount a volume to it with a persistent disc okay we have a Docker file so now what when you install Docker desktop that also installed the docker CLI which you can run from the terminal run Docker help to see all the possible commands but the one we need right now is Docker build which will turn this Docker file into an image when you run the command it's a good idea to use the T flag to tag it with a recognizable name notice how it builds the image in layers every layer is identified by a Shaw 256 hash which means if you modify your Docker file each layer will be cached so it only has to rebuild what is actually changed and that makes your workflow as a developer far more efficient in addition it's important to point out that sometimes you don't want certain files to end up in a Docker image in which case you can add them to the docker ignore file to exclude them from the actual files that get copied there now open Docker desktop and view the image there not only does it give us a detailed breakdown but thanks to Docker Scout we're able to proactively identify any security vulnerabilities for each layer of the image it works by extracting the software bill of material from the image and Compares it to a bunch of security advisory databases when there's a match it's given a severity rating so you can prioritize your security efforts but now the time has finally come to run a container we can accomplish that by simply clicking on the Run button under the hood it executes the docker run command and we can now access our server on Local Host in addition we can see the running container here in Docker desktop which is the equivalent to the docker command which you can run from the terminal to get a breakdown of all the running and stop containers on your machine if we click on it though we can inspect the logs from this container or view the file system and we can even execute commands directly inside the running container now when it comes time to shut it down we can use Docker stop to stop it gracefully or docker kill to forcefully stop it you can still see the shutdown container here in the UI or use remove to get rid of it but now you might want to run your container in the cloud Docker push will upload your image to a remote registry where it can then run on a cloud like AWS with elastic container service or it can be launched on serverless platforms like Google Cloud run conversely you may want to use someone else's Docker image which can be downloaded from the cloud with Docker pull and now you can run any developers code without having to make any changes to your local environment or machine congratulations you're now a bon ified and certified Docker expert I hereby Grant you permission to print out the certificate and bring it to your next job interview but Docker itself is only the beginning there's a good chance your application has more than one service in which case you'll want to know about Docker can POS a tool for managing multicontainer applications it allows you to Define multiple applications and their Docker images in a single yaml file like a front end a backend and a database the docker compose up command will spin up all the containers simultaneously while the down command will stop them that works works on an individual server but once you reach massive scale you'll likely need an orchestration tool like kubernetes to run and manage containers all over the world it works like this you have a control plane that exposes an API that can manage the cluster now the cluster has multiple nodes or machines each one containing a cubet and multiple pods a pod is the minimum Deployable unit in kubernetes which itself has one or more containers inside of it what makes kubernetes so effective is that you can describe the desired state of the system and it will automatically scale up or scale down while also providing fall tolerance to automatically heal if one of your servers goes down it gets pretty complicated but the good news is that you probably don't need kubernetes it was developed at Google based on its Borg system and is really only necessary for highly complex hi trffic systems if that sounds like you though you can also use extensions on Docker desktop to debug your pods and with that we've looked at 100 concepts related to containerization Big shout out to Docker for making this video possible thanks for watching and I will see you in the next one

So you're really digging this whole tech hobby and decide to wade a little deeper, running your own Netflix, AI assistant, or photo cloud on a server in your closet. Sounds awesome. Until you try to install more than one app on that server. That's because installing an app on a server isn't like installing it on a PC.

Here apps need dependencies, a specific config in a specific folder, a specific version of Python, specific specificity that makes you specifically scream. Install two apps that want different versions of the same thing and congrats, you've got a problem that needs a forum post from 2014 to fix. This is dependency hell and luckily it's a thing of the past thanks to a thing called Docker. But why does everyone use Docker?

What does using it look like? And why is it great for home labbing specifically? Let's dock in. Docker works by putting your desired app and all the stuff it requires to function inside a sealed box called a container.

You don't have to worry about trying to get two apps with different requirements to work. Everything is separate and because the app and its requirements all live in that one box, when you're done with it, deleting the container wipes the whole thing cleanly. So no stray files are left cluttering up your system. But that's not even the best part because Docker has been around for a while and has become a bit of an industry standard among developers, chances are someone has already packaged the exact app you want into a ready-to-run container.

So there's almost no setup on your end. After installing Docker, you literally just type Docker run and the app you want and a container with a working app is running in no time because building, running, and managing containers is done behind the scenes by the Docker engine. A lot of the home lab setups already install something like Docker engine, so it's already primed and ready to run containers. Maybe you want to run a media server or a web server or a remote VPN application.

You can quickly just do a Docker run command of those existing container images so that you don't have to bundle all those apps up and figure out how to get them on your host. Docker Engine can deploy containers so easily because of something called a container image. It's like a pre-made meal kit. The image has everything Docker needs to spin up a running container.

Docker Engine can also put a leash on each container to cap how much your server CPU and memory any one container is allowed to grab. That way, one badly behaving app can't devour all your RAM and drag everything else down with it. All of those ready-made recipes, i.e. image files, have to live somewhere though, and the most popular place is Docker Hub.

Think of it as the App Store for containers, except it's free and just about every self-hostable app is on it. But before you go installing 10 apps tonight, that convenience does cut both ways. Anyone can publish to Docker Hub, including people who probably shouldn't. You treat and trust Docker Hub very much like you would treat the source code on GitHub.

You've got to look at it, you need to check out the source, you need to trust it and inspect it and understand the risks of that particular piece of software before you want to download it and run it. Now, the nice thing about Docker itself is because it's an isolation engine, you can actually feel a little safer about running something you maybe haven't read all the source of. Worth noting though, Docker's isolation is real, but it's not a vault. Treat the sandbox as one layer of defense, not the whole defense, and vet the images from non-verified publishers.

Now, this all seems pretty revolutionary, and it is, but I'm going to let you in on a little secret. The idea of running sealed-off isolated programs not new. Docker didn't actually invent it. The blasphemy!

The truth is that Docker has an older, much heavier relative you've almost certainly heard of, the virtual machine. And the two get mistaken for each other constantly. We'll tell you how they're similar, but how they're different right after a quick word from our sponsor. Odoo, it's a business management software that has an app for just about anything your team could need conveniently in one place.

Whether that's a CRM tool for managing project workflow and invoicing, or even if you need help with email marketing, Odoo has tons of templates and ways for you to make sure you're reaching your audience. They'll even help you keep track of inventory with things like replenishment automation. You know what? That That sounds cool.

Click the link in the description for a free trial, no credit card required. Docker did not invent the idea of isolating programs. The broader idea is called containerization, and the virtual machine is where most people's mental model of it comes from. If you've ever heard someone called Docker a lightweight VM on a forum or Reddit thread, our expert would like a word.

Often containers are compared to virtual machines, but it's not really a fair comparison because virtual machines were really about emulating hardware so that we could run multiple kernels or operating systems on the same hardware. Docker doesn't really mess with running kernels. In fact, everything goes inside of the container except the kernel and the hardware drivers. The kernel is the core part of an operating system that actually talks to your hardware.

A virtual machine boots up its own entire operating system, kernel and all, which makes it powerfully isolated at the cost of being heavy. A Docker container skips all that and just borrows the kernel of the host it's already running on. So, unlike a VM that can take minutes to start and eat gigabytes of memory, a container starts in seconds and only needs megabytes to exist. To put it in perspective, it's not unusual to run 10 plus containers in the space a single VM would need.

And while some hardcore level labbers will actually run a few heavy and rock solid virtual machines and then run their lightweight Docker containers inside them for the best of both worlds, you probably don't need to go that far. If you're new to Docker, start with Docker Desktop or just download it on your Linux server where it will work natively. The cool part about Docker is that almost everything it's doing is really just low-level Linux kernel primitives. And the two that really enabled Docker to begin its journey as the default container runtime on Linux was two features in the kernel known as cgroups and namespaces.

Simply put, under the hood, Docker wraps features the Linux kernel had for years into a workflow that humans without a PhD in Linux kernel wizardry can use. But wait, if a container borrows the host kernel and that kernel has to be Linux, how does Docker run on a Mac or a Windows PC? It cheats, beautifully. [music] On a Mac or Windows machine, you can't just install Docker engine directly, but you can install Docker desktop, which quietly bundles a tiny Linux virtual machine and Docker engine so it can run using a working Linux kernel.

So, Docker on your MacBook is secretly running Linux. Don't tell anyone. One last thing because you're going to see it mentioned, Docker isn't the only container tool. Podman and Incus \{{}slash} LXC also exist and use basically the same commands.

But nearly every home server tutorial online says Docker, so unless you've got a specific reason to switch, I say keep it simple. And if running your own apps in steel little boxes sounds freeing, wait until you see what happens when you put your phone in charge of its own software. Go check out our video on sideloading. It ain't like it used to be.

think of Docker as a single shipping container it packages everything you need for your app to run now kubernetes it's the entire Port managing thousands of containers and deciding where they go how they run and what happens when something breaks but how do these two actually work together Docker is a platform that lets you build package and distribute applications in isolated environments called containers a container includes everything your app needs code runtime libraries and dependency think of it like creating a portable lightweight virtual machine but without the overhead this Docker file tells Docker to use a python image set up a working directory copy the app files install dependencies and run the app kubernetes is a container orchestration platform once you have containers kubernetes manages them across multiple machines it handles scaling load balancing and self-healing imagine running 1,000 containers across 100 servers kubernetes makes that possible this configuration deploys three replicas of a containerized app and ensures kubernetes keeps them running scaling them as needed Docker creates the containers kubernetes manages and orchestrates them you can run Docker containers without kubernetes but for large scale distributed systems kubernetes is essential all right let's break it down with more technical detail you already know Docker packages apps into containers kubernetes orchestrates those containers so how do they fit together Docker is the the building block its core role is to package and run applications in isolated environments known as containers the docker file is a text file that describes how to build your container it defines the base image for example python or nodejs installs dependencies copies files and sets commands for the container to run the docker image is the result of the docker file the image contains everything needed to run your app including code libraries systems tools Etc the docker container is a running instance of a Docker image think of it as an isolated process that can be run anywhere on any system that supports Docker Docker allows developers to create a consistent environment for their apps you can ensure your code runs the same way in development staging and prod kubernetes is about managing the scale it's a powerful orchestration system for Docker containers and provides features that Docker alone cannot handle at scale kubernetes group one or more Docker containers into pods and a pod is the smallest unit in kubernetes and shares the same network and storage which makes it ideal for tightly coupled applications that need to communicate with each other kubernetes uses deployments to manage the desired state of your app a deployment defines how many replicas of a pod should be running and kubernetes ensures that number is maintained so in short a deployment manages a set of PODS to run an application workload usually one that does not maintain State and this deployment provides declarative updates for pods and replica sets a replica set ensures that a specified number of pod replicas are running at one time so you as a developer describes the desired state in a deployment and the deployment controller changes the actual state to the desired State at a controlled rate and the deployment controller is a control Loop that watches the shared state of the cluster through the API server and makes changes attempting to move the current state towards the desired State kubernetes can automatically scale the number of PODS based on traffic load so horizontal pod autoscaling scales by adding or removing pods HPA starts new pod replicas to serve the additional traffic when demand decreases HPA scales the workload back down and HPA can Scale based on one or more metrics including HTTP request throughput message Q size CPU utilization memory utilization actual resource usage such as when a pod's CPU or memory usage exceeds a threshold kubernetes also automatically balances incoming Network traffic across pods Distributing requests so no single container becomes overloaded and if a container inside a pod fails kubernetes restarts it to maintain the desired state if a node crashes kubernetes reassigns pods to other available nodes the tldr is that while Docker runs the containers kubernetes handles the orchestration deploying scaling and managing hundreds or even thousands of containers across multiple servers it's the ultimate tool for running complex and distributed applications so here's a simple workflow of Docker and kubernetes developers use Docker to package the application into a container including all dependencies the docker image is pushed to a container registry like Docker Hub or Google container registry kubernetes pulls the image from the registry and deploys it as a pod then kubernetes will monitor the traffic and scale the pods horizontally adding more containers as needed it will also ensure the incoming traffic is routed to healthy containers and if any container crashes kubernetes automatically restarts it to ensure that the desired number of replicas is running the ultimate question is why use Docker and kubernetes together Docker alone is perfect for single container apps and local development environments but when you need to manage and scale many containers Docker alone falls short Docker and kubernetes is essential for production environments where automated scaling failover and management across multiple machines is critical so Docker itself simplifies building and testing apps while kubernetes ensures that those apps run reliably at scale together they're the backbone of modern Cloud native applications

one of the leading causes of imposter syndrome among developers is not knowing docker it makes it hard to go to parties where everybody's talking about kubernetes swarms shuffle sharding while you hide in the corner googling what is a container we've all been there at one point or another in today's video you'll learn everything you need to know about docker to survive as a developer in 2020 we'll take a hands-on approach by containerizing a node.js application i'll assume you've never touched a docker container before so we'll go through installation and tooling as well as the most important instructions in a dockerfile in addition we'll look at very important advanced concepts like port forwarding volumes and how to manage multiple containers with docker compose we'll do everything step by step so feel free to skip ahead with the chapters in the video description what is docker from a practical standpoint it's just a way to package software so it can run on any hardware now in order to understand how that process works there are three things that you absolutely must know docker files images and containers a docker file is a blueprint for building a docker image a docker image is a template for running docker containers a container is just a running process in our case we have a node application we need to have a server that's running the same version of node and that has also installed these dependencies it works on my machine but if someone else with a different machine tries to run it with a different version of node it might break the whole point of docker is to solve problems like this by reproducing environments the developer who creates the software can define the environment with a docker file then any developer at that point can use the docker file to rebuild the environment which is saved as an immutable snapshot known as an image images can be uploaded to the cloud in both public and private registries then any developer or server that wants to run that software can pull the image down to create a container which is just a running process of that image in other words one image file can be used to spawn the same process multiple times in multiple places and it's at that point where tools like kubernetes and swarm come into play to scale containers to an infinite workload the best way to really learn docker is to use it and to use it we need to install it if you're on mac or windows i would highly recommend installing the docker desktop application it installs everything you need for the command line and also gives you a gui where you can inspect your running containers once installed you should have access to docker from the command line and here's the first command you should memorize docker which gives you a list of all the running containers on your system you'll notice how every container has a unique id and is also linked to an image and keep in mind you can find the same information from the gui as well now the other thing you'll want to install is the docker extension for vs code or for your ide this will give you language support when you write your docker files and can also link up to remote registries and a bunch of other stuff now that we have docker installed we can move on to what is probably the most important section of this video and that's the docker file which contains code to build your docker image and ultimately run your app as a container now to follow along at this point you can grab my source code from github or fireship io or better yet use your own application as a starting point in this case i just have a single index.js file that exposes an api endpoint that sends back a response docker is easy then we expose our app using the port environment variable and that'll come into play later the question we're faced with now is how do we dockerize this app we'll start by creating a docker file in the root of the project the first instruction in our docker file is from and if you hover over it it will give you some documentation about what it does you could start from scratch with nothing but the docker runtime however most docker files will start with a specific base image for example when i type ubuntu you'll notice it's underlined and when i control click it it will take me to all the base images for this flavor of linux and then you'll notice it supports a variety of different tags which are just different variations on this base image ubuntu doesn't have nodejs installed by default we could still use this image and install node.js manually however there is a better option and that's to use the officially supported node.js image we'll go ahead and use the node version 12 base image which will give us everything we need to start working with node in this environment the next thing we'll want to do is add our app source code to the image the working directory instruction is kind of like when you cd into a directory now any subsequent instructions in our docker file will start from this app directory now at this point there is something very important that you need to understand and that's that every instruction in this docker file is considered its own step or layer in order to keep things efficient docker will attempt to cache layers if nothing is actually changed now normally when you're working on a node project you get your source code and then you install your dependencies but in docker we actually want to install our dependencies first so they can be cached in other words we don't want to have to reinstall all of our node modules every time we change our app source code we use the copy instruction which takes two arguments the first argument is our local package json location and then the second argument is the place we want to copy it in the container which is the current working directory and now that we have a package json we can run the npm install command this is just like opening a terminal session and running a command and when it's finished the results will be committed to the docker image as a layer now that we have our modules in the image we can then copy over our source code which we'll do by copying over all of our local files to the current working directory but this actually creates a problem for us because you'll notice that we have a node modules folder here in our local file system that would also be copied over to the image and override the node modules that we install there what we need is some kind of way for a docker to ignore our local node modules we can do that by creating a docker ignore file and adding node modules to it it works just like a git ignore file which you've probably seen before okay so at this point we have our source code in the docker image but in order to run our code we're using an environment variable we can set that environment variable in the container using the env instruction now when we actually have a running container we also want it to be listening on port 8080 so we can access the nodejs express app publicly and we'll look at port some more detail in just a minute when we run the container and that brings us to our final instruction command there can only be one of these per docker file and it tells the container how to run the actual application which it does by starting a process to serve the express app you'll also notice that unlike run we've made this command an array of strings this is known as exec form and it's the preferred way to do things unlike a regular command it doesn't start up a shell session and that's basically all there is to it we now have a full set of instructions for building a docker image and that brings us to the next question how do we build a docker image you build a docker image by running the docker build command there's a lot of different options you can pass with the command but the one you want to know for right now is tag or t this will give your image a name tag that's easy to remember so you can access it later when defining the tag name i'd first recommend setting up a username on docker hub and then do that username followed by whatever you want to call this image so in my case it would be fireship slash demo app and you could also add a version number separated by a colon from there you simply add the path to your docker file which in our case is just a period for the current working directory when we run it you'll notice it starts with step one which is to pull the node 12 image remotely then it goes through each step in our docker file and finally it says successfully built the image id and now that we have this image we can use it as a base image to create other images or we can use it to run containers in real life to use this image you'll most likely push it to a container registry somewhere that might be docker hub or your favorite cloud provider and the command you would use to do that is docker push then a developer or server somewhere else in the world could use docker pull to pull that image back down but we just want to run it here locally in our system so let's do that with the docker run command we can supply it with the image id or the tag name and all that does is create a running process called a container and we can see in the terminal it should say app listening on localhost 8080. but if we open the browser and go to that address we don't see anything so why can't i access my container locally remember we exposed port 8080 in our docker file but by default it's not accessible to the outside world let's refactor our command to use the p flag to implement port forwarding from the docker container to our local machine on the left side we'll map a port on our local machine 5000 in this case to a port on the docker container 8080 on the right side and now if we open the browser and go to localhost 5000 we'll see the app running there now one thing to keep in mind at this point is that the docker container will still be running even after you close the terminal window let's go ahead and open up the dashboard and stop the container you should actually have two running containers here if you've been following along when you stop the container any state or data that you created inside of it will be lost but there can be situations where you want to share data across multiple containers and the preferred way to do that is with volumes a volume is just a dedicated folder on the host machine and inside this folder a container can create files that can be remounted into future containers or multiple containers at the same time to create a volume we use the docker volume create command now that we have this volume we can mount it somewhere in our container when we run it multiple containers can mount this volume simultaneously and access the same set of files and the files stick around after all the containers are shut down now that you know how to run a container let's talk a little bit about debugging when things don't go as planned you might be wondering how do i inspect the logs and how do i get into my container and start interacting with the command line well this is where docker desktop really comes in handy if you click on the running container you can see all the logs right there and you can even search through them you can also execute commands in your container by clicking on the cli button and keep in mind you can also do this from your own command line using the docker exec command in any case it puts us in the root of the file system of that container so we can then ls to see files or do whatever we want in our linux environment that's useful to know but one of the best things you can do to keep your containers healthy is to write simple maintainable micro services each container should only run one process and if your app needs multiple processes then you should use multiple containers and docker has a tool designed just for that called docker compose it's just a tool for running multiple docker containers at the same time we already have a docker file for our node app but let's imagine that our node app also needs to access a mysql database and we also likely want a volume to persist the database across multiple containers we can manage all that with docker compose by creating a docker-compose.yaml file in the root of our project inside that file we have a services object where each key in that object represents a different container that we want to run we'll use web to define our node.js app that we've already built and then we'll use build to point it to the current working directory which is where it can find the docker file and then we'll also define the port forwarding configuration here as well then we have a separate container called db which is our mysql database process after services we'll also define a volume to store the database data across multiple containers and then we can mount that volume in our db container and hopefully you're starting to see how much easier it is to define this stuff as yaml as opposed to writing it out as individual commands and now that we have this configuration set we can run docker compose up from the command line which will find this file and run all the containers together we can mess around with our app for a little while and then run docker compose down to shut down all the containers together i'm going to go ahead and wrap things up there if this video helped you please like and subscribe and consider becoming a pro member at fireship io where we use docker in a variety of different project-based courses thanks for watching and i will see you in the next one

gee Barrette teaches us Docker containers and kubernetes fundamentals course for beginners ghee is a developer and trainer with more than 25 years of experience he is a Microsoft MVP frequent conference speaker and was the leader of the montreal.net user group for more than 23 years all this is to say he is the perfect person to teach you about Docker and kubernetes [Music] welcome to this Docker containers and kubernetes fundamentals training course my name is gibaret and I'll be your host in your Learning Journey Into The Amazing World of containers I'm a full-time trainer with a developer background and I'm based in Montreal Canada and that's where my strange accent comes from I'm certified on kubernetes and also on terraform Azure AWS and Google Cloud I'm very honored to be a Microsoft MVP in the Azure expertise and a digitalocean navigator you can reach me via the contact page and on Twitter what should you expect from this course you will not become an expert just by taking this course this is an entry-level course that will provide you with a strong containers and kubernetes Foundation and you'll gain enough knowledge to make sound decision at work or in your projects throughout this course you will find lots of Amazon activities to practice what you've learned you will use Docker and kubernetes locally on your PC or Mac so there's no requirement to have an account with a cloud provider are there any prerequisites for this course not really if you're a developer a devops specialist an ID Pro or even a technical manager that's totally fine no previous Docker or kubernetes knowledge is required we will cover a lot of ground you will learn about containers Docker and D Docker registry you learn about the kubernetes objects like pods workloads and services that's a lot of material and the goal here is to get you from zero knowledge to a kubernetes ninja well at least provide you with enough knowledge to Aspire being a kubernetes ninja I want to say a big thank you for learning Docker and kubernetes using this course if you like the course you can help me by making a small donation this is the link to my buy me a coffee page you can of course buy one of my other courses where you'll learn to run containers on different Cloud providers services and use a managed kubernetes cluster in the cloud and finally I wish you all the best in your Learning Journey [Music] let's see how to set up your laptop or PC for this course you need a laptop PC or Mac with either Windows 10 Mac OS or Linux if you have a Mac with an apple silicon most tools should run perfectly I will use Visual Studio code with the docker extension to help build create and run containers vs code is a free IDE that runs on Windows Mac and Linux on Windows and Mac you'll need Docker desktop with kubernetes enabled on Linux refer to the documentation on how to install Docker and kubernetes on your distro you'll need a Docker of account and a few easy to install tools refer to the setup instructions located below this video the lab files are located in a git repo on GitHub simply open this URL in a browser click on the code button if you have git installed on your machine you can type git clone with the link displayed here and if you don't have git simply click on the download zip button to download the code as a zip file [Music] let's talk about the microservices concepts if we head to Wikipedia and take a look at the definition that we find over there it says that it's a variant of the service oriented architecture or SOA a structural style slash architecture and that it arranges in application as a collection of loosely coupled services so instead of a large monolithic system we have multiple smaller pieces in a microservices architecture services are fine-grained meaning that each of them have their own responsibilities and the protocol use are lightweight like an API exposed over HTTP or grpc for example if we look at the monolithic architecture these systems were usually built as one single unit an IDE would group multiple projects and we would compile the whole thing as one single unit they were also deployed as a single unit so we would need to copy everything all the files on on a server and if we had to scale the system we had to spin a new VM and copy deploy the whole system on that VM and same for a third and a Ford server an example of such a monolithic architecture is a treaty application even though the system was clearly separated into layers it was all tightly coupled from the web project we had to make a reference to the business layer project and the whole system would run in the same address space with macro Services we break our big system into smaller parts each with its own responsibility so let's say we have a class that deals with identity in our business layer we can extract that code and place it in its own microservice we can then scale each of these smaller pieces independently from each other e's no strong bound since we expose there's no strong bound since we expose functionality through an API they can be written by smaller teams and each can use their own programming languages like go PHP C sharp and domain-specific data can be stored in separate databases so the way we would deploy a monolithic system is by deploying everything on a server all the dlls files needed to run the system we had to scale the deploy everything on more servers now let's compare that to microservices well microservices are deployed independently each can scale independently also need to scale back one service no problem so if you have an existing monolithic system how can you transform it into a microservices architecture well you need to break it into small units like the code that dealt with the identity in the in our business layer Martin Fowler author of the patterns of Enterprise application architecture book documented the way to achieve such a transformation using the Strangler pattern let's say our identity code is here in the Legacy system we can place a facade to Route the calls to it migrate the code and have the facade route the calls to the new macro service at some point as we go we will end up with less code in our Legacy system and when democracy and it's done we can get rid of the facade this pattern is very useful and you can learn more about it using the link in this slide and this concludes this lecture on the Marco Services Concepts [Music] let's talk about Marco Services anti-pattern because it's a thought Rosie I know it's kind of strange to talk about what can go wrong right away but I think it's very important first of all it's not some kind of magic pixie dust that you can sprinkle on top of a zigzing system and boom you get a beautiful Marco Services architect system it takes efforts and maturity to achieve this from one monolithic system you'll end up with a bunch of smaller pieces and that can add extra complexity a change to a microservice can have a domino effect and take your system down and what about securing all of these microservices it's also essential to use or introduce new processes in the organization like devops Ci CD and testing but be careful and don't try to implement everything at the same time it's a recipe for disaster take it step by step and make sure you have metrics in place to validate each of these steps and this concludes this lecture on microservices antibatterns [Music] let's talk about the microservices benefits and drawbacks since each microservice runs in its own address space there are less chances that if one of them goes down it takes the whole system down with it a microservice runs on open source Technologies so there's less Fender lock-in since they are smaller in most case they are easier to understand and that makes them faster to deploy and also easier to scale like we saw in the anti-pattern section there are some drawbacks complexity is added over mother's existence to resolve to her complexity issues so make sure your team is well trained and has made some proof of concept and make sure to start small adding one piece at a time testing might appear simpler since there are less functionality in a microservice to test but make sure to test the whole system deployment may appear simpler but one update can impact many Marco services and have a terrible domino effect ready to manage multiple databases calls between macro services will go through apis and this will add a bit of latency to all calls so make sure you test for that this key transition server will appear you'll make a call and it will fail or but try again 50 milliseconds later and it will work so make sure to implement some retry strategies in your code or by using a service mesh instead of one big point of failure you'll end up with multiple ones can your system survive if one microservice goes down and what about security are you okay for all these microservices can see and talk to each other so yes complexity is introduced for solving complexity issues and this concludes this lecture on the macro Services benefits and drawbacks thank you [Music] let's Now understand what is cloud native you may have heard the term Cloud native before but what it is exactly it's a way to architect and build complex systems taking advantage of modern development practices and the use of cloud infrastructure if we head to the cloud native Foundation website and look at the definition we see that it's quite a long one so let's break it into smaller parts Cloud native uses containers service meshes microservices immutable infrastructure and declarative apis we'll cover containers service meshes and macro services and the concept of immutability in this course but not how to build apis immutable infrastructure means that we usually never update something but we replace it with a newer version Loosely coupled systems mean that the functionalities are exposed through apis observable with the use of metrics creation and updates are automated and instead of making changes once every six months we deploy eye impact changes on a freaking basis and finally we use a series of Open Source projects to run our system when the cncf says to use open source projects they are not kidding this cncf landscape graph shows a ton of Open Source projects that you can use but don't worry you don't have to use them all the challenge really is to identify which one to use in the context of what you want and try to achieve and this concludes this lecture on cloud native head to the cncf website for more info [Music] let's go deeper in the cloud native Concepts Cloud native is about Speed and Agility the user wants new features right away without any downtime and the business wants faster release of feature to stay competitive a cloud native application architecture starts with clean code using domain-driven design techniques Markle services and kubernetes this course is all about microservices and kubernetes feel free to explore the concepts of clean code and DDD on your own with Cloud native we need to change mentalities infrastructure becomes immutable and disposable it is provision in minutes and Destroy as fast it is never updated but it's replaced with newer versions traditionally we would care about our virtual machines we would patch the OS update the apps with containers we create newer version with the software updates destroyed the previous running ones and replace them with the newer ones so the containers that you'll run will be more like cattle than pet of course this Cloud native thing is a lot easier when starting a new project a blank page or a green field however it's still possible with Legacy projects I really like the cloud native Foundation trail map because it breaks the journey to Cloud native into smaller measurable objectives you can set your own performance indicator to measure each steps to ensure a smooth Journey so let's take a look at the first steps your team must first learn how to cantonerize your application the developers and the IT Pros must know how to deploy and monitor containers you need to automate deployment through the use of continuous integration and continuous delivery techniques and tools you need to use an orchestrator like kubernetes and maybe deploy your application using L charts then you need to add observability so you can understand what's happening in your kubernetes clusters and be reactive use tools like service meshes to provide more functionalities inside your cluster Implement security through policies and wow these were just the first six steps Now understand that you don't have to implement all of this and especially not at the same time I really like this trail map because it breaks the journey into smaller steps that the management can understand and measure and this concludes this lecture on the cloud native Concepts [Music] let's take a look at the cncf website the website is located at cncf.io and from there you can take a look at the various projects maintained by the cncf information about on how to get certified Community Information like the conferences that the cncf organized each year so let's go back to the projects menu and you'll notice that projects are categorized in three categories sandbox incubating and graduated let's click here on this the second menu here and let's scroll down to the bottom and here you get the information about the meaning of these three categories and basically that's their maturity level sandbox projects are mostly newer projects uh well graduated projects our projects set conservatives Enterprises are more likely to use so let's take a look at the graduated ones we found here kubernetes Helm Jagger so let's click on kubernetes and basically that will show you the the project website let's take a look at the incubating ones here we find Linker D grpc let's click on grpc you can get more information about the grpc all right let's go back to this menu remember the trail map that I mentioned in the previous lecture well here it is cloud native trail map here's the nice diagram you can get more information you can get send that to friends colleagues and here's a link to the landscape diagram here it is it's super huge Let's uh let's click here on kubernetes and here we get very interesting information well you get the repository where the project is uh is stored you get the number of stars and the activity number of commits here you get you get the website address and also the Twitter handle here so you should follow the Twitter feed of the project that you're using so let's close this one let's click here on hell again website uh repository number of stars activity and the Twitter handle and this concludes this look at the cncf website thank you let's now talk about the containers Concepts containers containers containers they are everywhere but what are they exactly a container is a unit of deployment it contains everything needed for the code to run so the compile code the runtime system libraries and this is system tools you take a container push it on a server and it should run of course it can have some external dependencies like a database or a cache but the code deployed in it should run as is so why use containers because it is faster to deploy something small than something big like a complete monolithic system uh they use fewer resources they are smaller and since they are smarter you can fit more on the same server when using cicd techniques they are a lot faster to deploy you can run them anywhere and they are isolated from each other meaning that if one fails it will not take the whole system down with it so what exactly is virtualize let's compare virtual machines with containers a VM runs on some kind of Hardware where an OS is installed the OS hypervisor will let you create a virtual machine where you will install in OS so basically DVM virtualized the hardware and what's happening when a VM starts well you see the BIOS coming up and then the OS boots up and what about the size of that VM let's say we have a Windows Server VM it can take 12 gigabytes of RAM and 500 gigabytes of hard drive space and how long does it take to boot well depending on multiple factors something like 5 to 10 minutes now let's compare that to containers we still have the hardware and the OS of course there's a container runtime installed uh in the OS and containers images are run in memory now compared to a VM a container does not have to boot because it will use the host OS kernel this means that container starts in seconds because they don't have to boot they also use a lot less memory and hard drive space since there's no OS a small container can take a hundred megabyte of hard drive space and run in 64 or 100 megabyte of RAM so VM and containers virtual machine have a larger footprint they are slower to boot ideal for long running tasks container are lightweight they're quick to start they don't have to boot they're portable and they're ideal for short-lived tests because they you can spin one super fast so are containers replacing virtual machines our virtual machines obsolete absolutely no containers are just another tool in your toolbox and you need to find the right use case for them and also for VMS if you're old enough you must remember what the telephone booth is if not well before cell phones we used to make phone calls in these spoons by dropping a dime or a quarter anyways using a telephone boot analogy you can pack more containers on the same server than what's possible with virtual machines containers are made of layers you start with the base OS add customizations and add your applications let's take a deeper look at this screenshot the docker pull command retrieves and download a container image as you can see each layer is downloaded individually notice that each has a unique ID and that for the first ones Docker says that they already exist why is so Docker uses a local cache to store the container's images and if a layer already exists it will not be downloaded again the benefit is that if you pull version 2 of an image Docker will only download the layers not present in its cache one of the goal when creating container images is to create them with the smallest number of layers possible later on we'll see techniques on how to achieve that now can you write on these layers well no except for the top one because it is read right the lower ones are read only another concept is the container registry it's a centralized repository where you deploy the container images you create think Gita but for containers Docker as one called Docker up that provides public and private repositories and all major Cloud providers have container registry services the last container concept is the orchestrator an orchestrator allows us to manage scale monitor the containers that we run on our servers you can install your own or use a managed cluster offered by one of the cloud providers like AWS Azure Google Cloud we will come back to the orchestra Concepts after we have a better knowledge of containers and this concludes this lecture on the containers Concepts [Music] let's Now understand what is docker so what is stalker that may seem like a simple question but there's more to it there's Docker the company and Docker the platform document thinks the Mobi project an open source container runtime that follows the specs from the open container initiative doctors sold its Docker Enterprise division late 2019 to a company called mirantis so if you want to buy Enterprise support or get certified with docker you have to go through morentis Docker provides a container runtime that runs on Mac windows and Linux a command line tool to create and manage a containers a Docker file format for building containers and interestingly Windows lets you create both windows and Linux containers foreign if for some reason Docker doesn't seem to work on your machine try restarting it by using the restart menu from the system icon on Windows or by clicking on the debug icon in Docker desktop on Mac and windows and clicking on restart Docker desktop is very stable but I had some issues when my laptop was coming back from hibernation but that was a long time ago and I haven't had issues for a while and this concludes this lecture on docker [Music] the easiest way to run Docker on your machine is by running Docker desktop it's a free download available on docker.com so you click here on get started and you download the version for your OS so Windows Mac Linux here now if you're running Windows uh check the version of Windows that you're running if you're running Windows 10 version 2004 or a later version you can run what what's called windows subsystem for Linux version 2. so wsl2 basically it allow you to run a Linux distribution right into your Windows installation so Docker desktop can run its container by installing a virtual machine inside hyper-v or if you have wsl2 install it will it will install that virtual machine inside the Linux distribution and everything will be a lot faster so that's the preferred way so just to prove a point here I'm going to launch my hyper-v miniature and as you can see I'm not running any virtual machine so my Docker desktop installation uses wsl2 you'll find a link to this installation guide in the modules notes okay so let's take a look at the docker desktop I'm running Windows as you can see I can see my Docker desktop system tray icon here so for Mac User it'll be at the top of the screen of course I can right click on it and let's select dashboard all right so here I can see a list of um containers that are currently running I can stop them restart them delete them I can see a list of uh images that are installed on my machine here we'll come back to that later on there's a gear icon here that's the setting icon and in the general section here you can see that wsl2 is enabled so that's why Docker desktop doesn't use a Now preview virtual machine it uses the wsl2 to run the VM and there's the kubernetes menu here if I select it I can see that kubernetes is enabled so when you check that Docker desktop will download additional containers to run kubernetes right onto Docker desktop here so very useful there's a bug icon here which is the troubleshoot icon here so if at some point you're issuing Docker commands and the don't work or something's wrong uh running Docker Docker commands you can click here on the restart button here you can see my name here it means that I'm currently logged and if I right click down on the system play icon you can see that I'm currently logged in and I have the option to to sign out so what uh username and password did I use to to log in when you downloaded the uh the docker desktop you could have created a Docker account or a Docker Hub account so that's the same username and password so if you go to up.ducker.com and you logged in well that's the same account that is used here in Dr desktop thank you [Music] let's take a look at your first Docker CLI commands now throughout this course I will introduce you to various commands I will list them in what I call a cheat sheet list like what you find on the slide I will briefly explain what the commands are for and that will be followed by a concrete and Zone demonstration so when you install Docker desktop on on your Mac or PC it also installed the docker CLI tool our first command is Docker info this will display some information about the docker installation on your machine Docker version will display its version and Docker login will log you into a Docker registry by default this login command will log you into Docker Hub the registry from docker and this concludes this lecture on the docker CLI [Music] alright we need to open a new terminal window into Visual Studio code so let's select a terminal new terminal or you can use the shortcut Ctrl shift back tick key all right this will open a terminal window and let's take a look at the commands that we will run they're really basic it's just for testing that our Docker installation is working correctly so let's type Docker info this will give me some information about my current installation what's happening I have 47 containers three stuff 44 running 25 images and so on and so on so information that that is quite useful for debugging purposes I can see that the virtual machine running that running Docker desktop is running as to CPU it has two gigabytes of memory allocated all right sounds good let's uh type now Docker version and this will uh give me some information about the version number of different parts of Docker desktop so again useful for debugging purposes these are not commands that you will run on a day-to-day basis but they're quite useful for troubleshooting the last command is Docker login so I'm just going to type Docker login without any username password see what's happening and well the command says that I am login successfully why is so well if I right click here on my Docker desktop I can see that I'm already logged in so that's why I didn't have to input or type any username password I was already logged in [Music] let's now see how to run containers the docker pull command lets you download an image from a registry Docker run will execute an image in memory as a container if the image is not present in the docker local cache it will be downloaded automatically using run with the Dash D flag will run the container in the background giving you your comment prompt or terminal back the start command will run a container in the stopped state Docker PS will list all the containers currently running and add the dash a flag to also lists all the stopped ones Docker stop will stop a container running but the container will still be in memory we will see how to remove them from memory in a few minutes Dr kill will well kill a container that might be stuck in memory you usually don't use this comment but it's useful to know Docker image and spec will give you some information about an image very useful for debugging purposes so you may notice that we have two parameters here one is called image name and the second one container name so what's the difference the image name is the name of the image as you find it in the container registry and the container name is the name of the running container so you run an image using its name and then interact with it using the running instance name the Run come in as an optional flag call dash dash name that lets you specify your name if you don't specify one Docker will Auto generate one for you you can set limits on the memory and the CPU that the container can use when using the Run command so how do you run a container using the docker run command you specify the image name as found in the container registry you specify a name for the running instance and with the published flag you map a port from your local OS to the port that the container is listening to you can list the running containers using Docker PS notice how we stop the container by using the running name and not by using the image name then we remove it from memory using the remove or RM command containers are not black boxes you can attach a shell to your terminal and run commands that will execute right inside the running container by using the dash it switch and the name of the program you want to run a Windows container well you can run Powershell using the docker container exact command you can attach to a running container here's a screenshot showing the docker run command and notice the terminal problem changing when attached to a container so how do we clean up things the remove command lets you remove a container from memory but first it must be in the stop state for the command to work here's the remove command getting a list of the stopped container and removing them all the images that you pull will be cached locally you can get a list of these images using the docker images command use the remove image or RMI command to delete an image from your machine after a while you may end up with a bunch of images to do some spring cleanup I use the system prune command this will delete all the image currently not in use so be careful using this command and this concludes this lecture on the docker CLI [Music] let's now run our first container we'll run an engine X web server so something pretty basic that'll be perfect for this this lab all right let's open a terminal so terminal new terminal or Ctrl shift back tick perfect so we'll run disk command so let's take a look at the command first so I'm going to run a Docker run and let's go at the end of the command this will be the image that will run so an nginx image we'll give it a name so the name will be web server that will be the name of the running instance will map local All Sport 8082d or that the container is listening to so port 80. n d 4G decimal so we can get our Command Prompt or terminal prompt back so let's run this ha something interesting is happening unable to find image nginx latest and you see that the docker as pull all the different layers locally here so now if I issue a Docker PS it will list the containers that are currently running so I have uh three containers running so the kubernetes dashboard and the metric server here so don't uh look at these let's focus on this one so this is the containers it is that we just launched the nginx image here it started 30 seconds ago we can see that the port 8080 local levels is mapped to the port 80 here and the name is web server so let's launch a web browser and let's type localhost 8080 the container is actually running fantastic all right so we can get a list of the images installed on your machine here by using Docker images so I have a bunch I may have a lot more than you but let's focus on the last one here uh nginx the tag is latest image ID and this is the uh the size all right fantastic let's try to connect to uh to it so we'll issue a Docker container exact we'll give it the the name so the running instance and the program we want to run so look at the The Prompt now root at some some ID so the ID is the actual container ID so I'm logged in as root on that container so now I can issue some some commands so let's do in the last let's thy fell less been to see what's in there so different commands so I'm connected to to that running container that's pretty uh pretty cool so I can issue commands look at the logs if any and do some troubleshooting so this is super useful for debugging purposes uh we'll use that a lot in the various steps that we'll do together all right let's get out of there by typing exit all right you can just clear my screen here so our container is running how do we stop it we use the docker stop command but but but look at the the parameter that we use the name of the container that we use we use the running instance name not the name of the image so that's pretty important here Docker stop web server all right but the container is still in memory if I do a doctor PS it's not listed anymore as you're running container but if if I type darker es Dash a ha for all I can see that my my container is still in memory here so I need to remove it from uh from memory so we'll use the docker RM and the name of the running instance so RM for remove and now the container is no longer in memory awesome now now now the container is still well the image that was used to create the container is still on my machine so if I type darker images you see it's still it's still here so that takes 133 megabyte of this space if I want to get rid of that I use the RMI so remove image command and the name this time of the image not the name of the running instance because none are running right now and you see all the layers have been deleted [Music] let's now see how we can build containers Docker build that you create an image using a Docker file if you run the command in the same folder where the docker file is located simply use a DOT as the file name if the file is located in a different folder specify the location using the F flag the tag command let you assign a name to an image this tagging has two parts a name and a tag the tag is usually used to specify the version number so what is a Docker file well it's a text file listing the steps to build an image here's the simplest Docker file I can imagine two lines the from command specified the base image when building new images you always start from something already existing in this case an image with the nginx web server using the Alpine version and then the copy command copies everything from the current folder to a folder inside the container using the build command we create a new image specifying the docker file remember to use the dot when the file is located in the same folder here's another one this time a little bit more complex it is used to create an image running a node.js app let's take a look at it the from command specify the base image using the Run command we run the package manager inside the container to install node.js next we copy all the local files into a folder named SRC inside the container we use the Run command to do an npm install we then had some metadata in this case we tell the container to listen on port and finally we tell the container what to run when starting so as you see this Docker file contains the stats needed to run our node.js app we saw what tagging was a moment ago let's explore this again using the docker tag command we name an image using a name and optionally a tag if you don't specify a repository name it will default to Docker Hub later on we'll see how to push images to different repositories and we'll have to specify the Ripple's Journey name when tagging our images and this concludes this lecture on how to create Docker images [Music] let's see how can Visual Studio code help us build and run containers so what is visual studio code and why talk about this tool in this course well it's a text and code editor it's free and open source it runs on Windows Mac and Linux and you can download it for free using this link so you will work a lot with text files creating Docker files and later on Docker compose and yaml files a tool like vs code will help you because you can install plugins that will make your life easier using a different text editor no problem in vs code you can install plugins by clicking on the extension icon in the left menu then you search for Docker and install the extension from Microsoft the extension lets you add Docker files to your projects using the command palette open it using the view menu or type Ctrl shift p type Docker add and select add Docker files to workspace the extension will ask you a few questions and we'll create the docker files for you vs code has a built-in terminal where you can type commands or you can run commands using the command palette here's another example when running a container and when using the command palette there's no magic there the extension will simply issue a command in the terminal but sometimes it's a great way to learn creating Docker files is okay but what I like the most is the UI provided by the extension helping me manage my container so if you click on the docker icon you can see the images installed on your computer and you can even see the containers currently running right click on an image to manage it same thing for the containers currently running very very very useful and this concludes this lecture on vs code thank you let's now use Visual Studio code to containerize a node.js express application so I already installed the docker extension in Visual Studio code but let's take a look at it so if I click here on extensions and let's search for docker here it is that's the one from Microsoft almost 6 million install at the time of recording so that's the one all right so let's go back to our files so I have my node.js application here if I click here on package.json [Music] the name of my application is my Express app so that'll be important in a few seconds all right so let's first add the docker file to our project so we'll use the tooling provided by the extension to do that we'll go to the view menu command palette or you can use the shortcut Ctrl shift p right and we'll type Docker add and there are two options here Docker files or Docker compose files so we haven't looked at the docker compost file so let's select the first one Docker files the extension hack is asking us about the application platform so it's a node.js application but it can generate Docker files for net python Java C plus plus go and Ruby apps so let's select node.js all right and next next it's asking us where is the package.js file so here it is it set the root of my application so I'll select this one the port that the application is uh listening to 3000 perfect and do I need the optional Docker compost file no not at this time all right so the extension quickly added a the docker file and also the docker ignore file and also a folder for vs code so let's take a look at the docker ignore basically it's a list of files not to deploy in our container and the docker file has been created for me to use a node base image it copies everything into the Container the image and expose port a portrait 3000 perfect so let's now build this image so we'll go back to the command palette view command palette Ctrl shift p and this time we'll search for Docker build all right here it is Docker images build image all right so I just issue uh the command and look look at what's happening the extension is not a black box uh issuing some crazy or strange uh commands in the background it's just issuing a a Docker Bill command here and remember a few seconds ago I mentioned that the name of the application in the package.json file is is important well that's the name of our uh of our application here so the extension use it to generate the name for my image all right it's just a Docker Bill command nothing fancy but using this extension is a great way to learn about some uh some commands that you may want to use at a later time from the terminal prompt okay terminal will be reused by test press any key to close it let's press any key all right the image has been built let's run it now so view command palette okay Doctor run so we'll use the first one and uh there's a list of my images and here is my Express app perfect run that select image that's the latest the tag latest perfect and look at what's happening Docker run sport 3000 to the port that the container listening to so nothing fancy here is uh it's a comments that you can type yourself but uh the extension to that are for you so if I start my browser local ocean I go to Port 3000 there it is so that worked perfect let's now close this let's now use the UI provided by the extension so in the left menu let's click on the docker icon here and from there you have a list of the running containers the images and the different Registries that you connected to all right so here's our Express hat my Express apply it is and it's running name the instance name is called magical underscore Direct uh why is so well when we issued a Docker run command using the command palette the extension didn't provide a name so uh one was generated automatically by by docker so from there I can view the logs attach shell inspect open in a browser so if I select that there it is that's my app listening on Port 3000 uh I can stop it restart it remove it so let's uh let's remove it here so are you sure you want to remove container yes it's not in memory anymore it's not running anymore but if I look at the list of the images that I have here it is my Express app so I can right click on it and I have a few options I can run it inspect it pull push tag and even delete it so let's let's run run it okay and you see at the top my Express app app app and look at the name gifted underscore El beckyan because if you look if you uh inspect the docker command that was issued by the extension there's no name provided so Docker generated one for us all right so let's close this we can stop or remove it from memory and if we no longer need the image we can just remove it here are you sure you want to remove the image blah blah blah okay let's remove it and it's gone so what's happening in the background is that the extension is simply issuing a Docker remove image RMI to to remove the image from from this so nothing nothing really fancy the extension it's not a black box that is issuing some strange commands it's just regular and Docker commands but I really like the the UI provided here and so instead of uh typing uh okay let's uh let's try this let's um um um let's clear this and let's type Docker images here's a list of my all my images but here I have a nice UI listing all the the same images here so sometimes it's it's easier having a little UI to help you accomplish some tasks other times it's easier issuing the commands from the terminal or the command prompt so it's up to you you can use the UI or The Terminal thank you [Music] let's talk about data persistence containers are ephemers and stateless so you don't usually store data in them of course you can write data in a container but if you destroy one or if it crashes any data stored in it will be lost so it's okay to write some log files or scrap data that you don't want to keep as long as you understand that you will lose these files at some point in time to purchase data you need to store it outside the container in what we call a volume a value Maps an external folder or even a cloud storage service to a local folder inside your container so your app sees a volume just like any regular folder the OS in this diagram represent the server or the virtual machine where the container is running as you can see a local folder is mapped to the VM file system so they are stored in a volume where survive a container restart or crash there's still a chance that we can lose the data if the VM crashes so later on we'll see how we can use some type of external storage provided by the cloud provider but first thing first in the next lecture you'll see how to create a volume that maps to a VM file system and this concludes this structure on data persistence [Music] um let's see how to create volumes here's a cheat sheet listing the docker commands for managing volumes Docker create volume will create a new volume LS will list all the volumes volume and spec will get you information about a volume remove will delete a volume destroying all the files storing it and volume prune will delete all the volumes currently not mounted or not in use use use so be super careful careful using this command alright you first need to create a volume using the docker volume create command then when you run a container you need to use the V switch or to volume a parameter specifying the volume name a column and the name of a local folder that folder will be a logical folder in your code so your code will see just like any regular folder if you use the inspect command you'll see The Logical folder location in DVM instead of using a volume you can specify a local folder this is great for testing purposes let's say you started developing your service and that you want to test on your Dev machine if your code can read and write some files correctly you can use this kind of mapping but don't use that in production using the inspect command you can see the local folder path and this concludes this lecture on volumes [Music] a software system data outside of a running container so to do that we'll use a value so I'll open a terminal and we'll use the docker volume create and the name of the volume so this will create a volume perfect let's Now list the volumes on my machine so there's a few here the first four were created earlier this is the volume that we just created now let's run a ninja next image and let's attach a or Mappy a local folder to that volume so we'll use Docker run Dash D for detach we'll name our instance fault test with the Dash D for volume we'll use the volume name that we created earlier and map that to a folder called ATP on our nginx image all right to that okay excellent so that worked now let's connect to our running instance so Docker exec Dash it the name of the instance and it will run bash here perfect let's first do an LS to see if we see the app folder there it is so that's that folder is mapped to the volume so anything that we store or write in that folder will be written externally outside of the container so that will persist all right uh just for fun instead of doing a cat Let's uh install Nano inside the running instance so I'll do first in apt-get update remember this is uh this is running inside the the container and we'll install Nano app get the install Nano which is a small editor perfect LCD into the app folder in open Nano so Nano and we'll create a file called text test.txt hello volume [Music] all right and we'll use Ctrl o to write to to disk Ctrl o enter and control X to exit Nano perfect if I do an LS here I should see my file there it is test.txt so let's exit the running instance and what we'll do will stop the running instance and we'll remove it from memory stop it here and we will remove it from memory with our m okay so now the container or the instance is gone if we would have stored some data inside that container it would it would be lost at this point but we use a volume so the data was stored externally so let's try to create a second instance same thing using the same volume and let's exec to it perfect let's see what is in the app folder here is our file so let's do cat test.txt just to prove hello volume that work let's exit here this proved that by using a volume your data persists a container restart or crash here the data is still there until I remove the volume so if I issue a Docker volume removed and the name of the the volume you see doesn't work why the error says that the volume is in news interesting so I need to stop any container instance that is that is running and remove it from memory before deleting or removing a volume let's try uh again term Docker volume RM for remove by volume and this time it worked now another thing I want to show you let me create it again perfect and let's switch to the docker UI here if I click Docker here there's a section that list of the different volumes here so my vowel is listed here so here I can inspect I cannot look at the files but I can I can manage it so I can click here on remove that will delete the volume are you sure yes [Music] let's now see the yaml concepts yaml stands for yaml ain't markup language it's a way to serialize data so that it's readable by human beings it's the file format used by Docker compose and kubernetes here's the sample yaml file for a key value pair you specify the key a column data space and a value don't forget the space it's mandatory here are some nested values you specify child values using two space indentation and quotes are not needed for string values here's a list again the child elements are indented with two spaces and there's a space after the dash this is what we call the block style there's also a flow style that looks like Json so you may be tempted to use it if you're familiar with Jason but don't I never saw any sample or any documentation using this flow Style since it's easy to forget a space and you can spend quite some time figuring out why your yaml file doesn't work you can use tools like this linter available on yamlin.com it will parse your yaml and flag any errors very useful and this concludes this lecture on the yaml concepts [Music] let's now take a look at the docker compose Concepts let's say that your app is composed of multiple containers you run the front-end container using a Docker run command Docker run again for the backend container and again for the redis cache container so you end up issuing multiple Docker run commands to run your app would it be nice if you could deploy your app using one single command well that's the docker compost go to Define and run multi-containers application using a single yaml file there's a compost plugin that extends the docker CLI and let you run those Docker compose files these specifications are available here if you look at Docker compose before you may have seen that sometimes the commands are using an iPhone so Docker Dash compose and sometimes they do not why is this at the dockercon conference in 2022 Docker announced the general availability of couples version 2. okay this means that there was a V1 before and the V1 command line tool was installed separately from the docker CLI it was built using python so you needed to have python installed to run compost V1 V1 V1 and the syntax was Docker Dash compose couples V2 is a drop in replacement meaning that all the V1 commands are working as expected it's installed as the docker CLI plugin automatically by Docker desktop to use it you type Docker space compose no hyphen needed here it's written in go so no need to have python installed to run the command in summary it's simply a faster version of the docker compose tool shipped as a Docker plugin instead of a python application here's a Docker compose file there are three containers defined in it web API 1 web API 2 and API Gateway the name that you use here defined the network or host name for that container the code running inside your container can use these hostname to communicate between each container for each of them you specify the image to run you set the internal and external Port the container will listen on note that the API version is now optional so it's okay to skip it you may ask yourself should I use darker compose or not Docker compose is perfect for small workloads that don't require a full orchestrator like kubernetes it's perfect When developing and testing locally before deploying to kubernetes Some Cloud providers offer services that support Docker compose like app service on Azure and ECS on AWS and of course you can simply use a virtual machine or a VPS virtual private server with the digitalocean or linued and this concludes this lecture on the docker compost Concepts thank you LaSalle used Docker compose here's a cheat sheet listing some of the docker compose commands Docker composed build lets you build the containers as defined in your Docker compose yaml file if the file is located in another folder you can use the optional Dash F per meter and specify the files location start we'll start all the containers as defined in your yaml file step will stop them but they'll remain in memory up we'll do a build followed by a start this is super Endy use the Dash D parameter to run the command in the background and take back your terminal prompt PS will list what's running remove our RM we'll remove the containers currently from memory down we'll do a stop followed by a remove again this is super ND logs will display the logs for a container and you can open a session inside a container by running Docker compose exec the container name and the program to run the docker compose file is located inside a folder and if you run Docker compose up this will launch your application if you try to run a Docker compose up a second time from the same folder nothing will happen because the application is currently running if you want to run a second instance of your application it was impossible with Docker compose V1 but with V2 you can use a project name to launch a second instance of your application from the same folder here's the cheat sheet for some of the new commands Docker compose dash dash project Dash name followed by a project name this will run a then sense of the application as a project the shortcut is much shorter to type Dash p instead of that Dash project Dash name you can list the project currently running by using compose LS CP will allow you to copy files from the containers so this is super ND to retrieve let's say log files and you can copy files to The Container so from your machine your desktop or laptop to The Container by using Docker compose CP The Source pad the container ID and the destination path here's an example imagine that the docker compost file is located in the same folder where you run these commands you simply use the up command to build and run the containers and to take them down simply use the down command and this concludes the structure on Docker compose [Music] in this lab we will deploy a Docker compose application let's take a look at our Docker compose.yaml file we have one section called services and under that section we Define two Services the first one is called Web Dash Fe F4 front end it's a python application and instead of using an image from Docker up will be building that image using the build parameter the dot means that the docker file is at the same level as the docker compose file so here it is it's a simple python application just one file app.pi and requirement.txt that we're copying on that base image it will be listening on Port 5000 and this is our second service it's the redis cache and this time we'll be using an image from Docker up all right let's open eight terminal and let's build the image using Docker compose build perfect my image was build now I can launch the application using Docker compose up and Dash D for detach now I could have skipped the build step because up does a build first and then a start so it's super ND so let's use Docker couples up and my application is up and running I can test it if I go to local OS 5000 you visited me one time and do a few refreshes five time perfect okay I can list the running containers using Docker compose PS I can also use Docker PS since it will list the docker container currently running and I can look at the logs for my front-end service using Docker compose logs Dash F and the name of the service and if I move that a little bit and it F5 a few times you see new entries are logged perfect that works let's do a Ctrl C to terminate the log streaming and we'll use Docker compose LS to list the currently running projects I have one project running it's called l09-04 Docker compose well basically when I use Docker compost up I didn't specify a project name so Docker compose use the folder name as the project name now let's try to create a second instance of our application if we use Docker compose up Dash D again well Docker compose tell me that the application is running so it will not start a new version we can try to deploy our second version using a project name so Docker compose Dash p for project name we'll name it test up Dash D let's see what will happen starting oops we have an error here hmm bind for local OS 5000 failed Port is already allocated of course are my local old sport 5000 is in use right now so what I need to do I need to change that Port here the localhost port I'll use 5001 save the file and use the same command Docker compose Dash B project name test up Dash D and this time it worked awesome let's open our browser and let's go to Port 5001 yes that worked a few refreshes 10 times let's go back to Port 5000 the first instance 15 times all right so let's do a Docker compose LS again to list our projects now we have two projects the first one that we deploy without specifying a project name and the second one with the project name test let's delete our for instance by using Docker compose down so I didn't specify a project name Docker compose use the folder name as the project name now if I do the docker composer list I should have only one project running yes it's test now I can delete that one using Docker compost specifying the project name and down let's list the projects again Docker composer less nothing running PS listening the containers nothing and why not try Docker PS nothing in memory [Music] in this lab we will deploy a Docker compose sample application that is composed of three services a web front-end build with react nodejs backend and the Maria DB database let's take a look at our Docker compose file this is a more complex than what we saw so far so let's try to break it into smaller pieces here we have the definition of our tree services if we start from the left we have the back-end service top right the database DB and the front end now if you look at the DB service you can see that we're referencing an image that will pull from Docker hub and the other two Services we're using the build parameter this means that we will build these two images looking at the backend service we can see that we specify build and context contacts with the value of backend backend is actually a subfolder where the docker file is located next we're defining two networks public and private we can see that our front-end service is using the public network the backend service is using public and private and the database service the DB service use only the private Network front end being in the public network cannot communicate directly with the DB service but backend being in both public and private can communicate with both DB and front-end services we're defining also two named volumes back in dash modules and DB Dash data and we can see highlighted in yellow that our DB service is using DB data and our backend service is using back-end modules highlighted in yellow we also see that we're using other volumes these volumes are scoped at the service level and are not shared between services to create an instance of our Docker composed application we simply use Docker couples app and to bring it down Docker compose down in Visual Studio code let's take a look at the docker compose file so it's called compose.yaml here we have our services section Network section where we Define two networks the name volumes back-end modules and DB data and secrets that we haven't seen yet where we're defining one key value pair so the key is DB password and we get the secret from a file the file is located in the folder called DB and it gets the value from a file called password.txt if we open the DB folder here's the file with the value the secret and that will be injected when we run Docker compose up okay let's go back to the services section and here we have our backend DB and front-end Services let's take a look at backend we're using the build a directive and we are setting the context to backend it points to this folder called backend here and the docker file is located inside that backend folder with the application same thing with the front-end service build context front-end and we're getting the docker file and the application from the front-end folder and what about the DB service well we're using an image that will pull from Docker hub okay let's build our two images we'll open a terminal and run Docker compose build our images were billed perfect now let's run the application by using Docker compose up Dash D ER perfect the application is this thing on Port 3000 so let's open a browser and type local OS 3000 is our react application it's working awesome we can list the containers that are currently running by using Docker compose PS we should have three backend DB front end awesome let's take a look at the logs from the back-end service Docker compose logs Dash F backend and these are the logs for our backend service awesome we can type Ctrl C to stop the log streaming and we can take our application down by using Docker compose down this will stop and remove the containers from memory do we have something else in memory we should not perfect and even if I type doctor PS there's nothing however the volumes are still there when you do a doctor compose down it will remove the containers from memory but will not remove the volumes so if I open the docker desktop application and I click on volumes here I can see that I have a few well three volumes that were created a few minutes ago ago ago so I need to delete them manually I can select them right here and click on delete or I can do the same thing by clicking on the docker icon in vs code locating the volumes here and delete them find it easier to do that in Docker desktop because we see when the volumes were created so I'm pretty sure that these three are the ones that I need to delete I'll click on the delete button and confirm thank you [Music] let's take a look at some of the compose file features it's a good practice to set limits on the resources that your container will use use use in this example I light it in yellow we tell Docker to start the container with a quarter of a CPU and 20 megabytes of RAM RAM RAM the green section is the limits that we are allowing in this case half of a CPU and 150 megabytes of RAM to set an environment variable that will be injected in the running instance simply set the key value pair in the environment section those values can be overridden at the common line using the dash e per meter you can reference an environment variable using the daughter curly bracket syntax this way you can set the variable on your machine or server and use it directly in the compose file you can place the values in a file that you will name dot EnV located in the same folder as the compose file the compost command will automatically read the values from that file by default all containers specify in a compose file will see each other using their service names here we have two Services web and DB the code running in the web service can communicate with the second one using DB as the hostname and vice versa the web container is visible from outside of the docker network using the port number configured in the left portion of the ports value web is listening inside the docker Network on Port 80.

DB can reach web on Port 80. finally DB only exposes one port number that's the internal port web can reach DB using Port 5432 but DB is not visible from outside the docker Network if you have a compose application with multiple containers you can restrict who sees who by configuring networks in this example we're defining two Networks front end and back end proxy can see app because both are part of the front-end Network however proxy does not cdb because it's not part of the backend Network when using multiple containers you may want to start some of them first and wait until they are running before starting the other ones a typical use case is a database that you want to run before sorting the main application doing so is easy using the depends on parameter where you simply specify what is the service name that the service is dependent on in this example app depends on DB so compost will first start DB and when DB is running compose will then start app you can declare volumes in the volumes section these are called named volumes and they can be used by all these services that you are declaring in the compose file to use a volume from a service map it to a local folder using the volume name colon and the virtual path inside the container optionally you can make the mapping read-only by appending colon Ro to the mapping you can also create a mapping without using a named volume this mapping can't be shared across services it's also a good practice to set a restart policy let's say that you deploy your couple's app in a VM and at some point you need to install some OS batches and you need to restart or reboot the server what will happen to your compost app well if you don't specify a restart policy the one by default is no meaning that compose will not restart the containers if they were shut down by reboot you can set the policy to always this way compose restarts the containers until their removal on failure resource a container if the exact code indicates an error and lastly unless stop does a restart unless you stop or remove the containers and this concludes this look at some of the docker compose features [Music] let's now talk about container registries so what is the container registry it's a central repository for container images you build an image locally then you push and store the binary the different layers 2D Repository they can be private or public the default one is Docker hub Microsoft AWS and Google each offer container Registries as service the benefit of using a repository from your cloud provider is that the images are located near your app so no apps over the internet to retrieve the images so let's say we want to retrieve an image from Docker up we issue a Docker pull command and Docker downloads the images layers and store them in its local cache and this concludes this lecture on container registries [Music] let's see how to push and pull images to Docker up make sure you are logged in with your Docker user account to be sure simply type Docker login without a username and password Docker will tell you that you're already logged in if not enter your Docker username and password you need to tag an image with the repository name by default it's your username if you have created some organization in Docker hub prefix it to the name of the image in this example I want to push this image to my Kates Academy organization then use the push command and don't forget to specify the organization name it's part of the image name to retrieve the image we use the pull command with the image full name on Docker up public images are available for download to anyone if you don't want to share them you need to create a private Repository later on we will create one using our cloud provider and this concludes this lecture on Docker up [Music] let's now push our first image to Docker hub hub hub first thing first let's make sure that we can log in into Docker Hub so if you head to up.docker.com make sure you can log in also if I right click here on my Docker desktop icon I can see that I'm logged in in in perfect we will containerize a node.js Express application so we'll first add the docker file we'll use the tooling so view command palette if you can type Docker add and Docker files to workspace this is a node.js application the package.json file is located at the real so that's the correct one it's listening on power 3000 and we don't want Docker compose files perfect now we need to build the image so let me open a terminal perfect and we need to issue a Docker build command with the Dash D for tag parameter but notice here we need to prefix the uh the name of the image with our registry name if I select this I need to prefix that with my name here the name of my registry let me run that the image was built successfully now I need to use the push command to push that image onto Docker Hub again I'll select this and let's replace the registry name with mine use your home and see what's happening here Docker is pushing each layer to my Docker app account here all right no errors things went fine let's go back here I'm gonna refresh this page and here it is here's my um uh image here my Express image so I can click on this I can edit the um um um the information I can see that I have my tag is V1 right I can get more information here I can here click on public view so by default the repositories on Docker Hub are public anyone can download and view and download your images this is the view for someone who would look at my image so with the pull command Docker pull let me go back here I can see the tags and so on and so on and if at some point you want to delete this you go into the settings tab here you scroll down and you can delete that repository here all right let's go back now let's try to pull that image to our computer here the first thing uh I will try to do is remove it from my computer let me type this RMI so remove image the image has gone completely so now let's try to pull it from uh from Docker uh okay Perfect Pull complete my image is is back here now now now you see that I use the V1 tag here to to tag my image with the with the version number so let's try to build a version 2 of that image let me copy that and again let's replace this part this placeholder with my registry name image has been built so let's now push it it it two Docker uh same thing as we did before but this time we're pushing version two okay let's go back here let's go back to my Express or the general tab look here I have V1 and V2 if I click here on tags I can see when the image was was pushed tags and people can download V1 or V2 if I want to remove my images I use the RMI command so remove image I'll remove V1 and also I remove V2 and on Docker up if I no longer need this Repository I click here settings scroll down a little bit click on delete Repository and I need to enter the name of my repo just to make sure and click on delete and now it's gone [Music] time to introduce kubernetes so kubernetes or also known as Kates so the letter K followed by the number eight so eight letters and then the number s and it's pronounced Kate's so kubernetes is a project that was created at Google version one came in July 2015. it was the third generation of container scheduler from Google previous projects were Borg and Omega and Google donated the kubernetes to the cncf so now the development is supervised by the cncf is currently the leading container orchestration tool it's designed as a Loosely coupled collection of components for deploying managing and scaling containers it's vendor neutral so it's not attached to a single company and it runs on all Club providers and there's a huge community ecosystem around kubernetes so what kubernetes can do service Discovery load balancing it can bridge to the cloud providers storage services can provide rollout rollbacks capabilities and can monitor the health of the containers can manage configuration and secrets and the same API is available either in a on-premising solution or in every cloud provider so what can't kubernetes do it can deploy or build your code and it does not provide application Level services like databases service buses caches here's a quick look at the kubernetes architecture this diagram was taken from the kubernetes documentation we'll take a closer look at each component but for now let's just say that it's composed of a master node also called the control plane so that's the portion to the left and the control plane runs the kubernetes services and controllers and you have the worker nodes these node runs the containers that you'll deploy in the cluster so a container will run in a pod a pod runs in a node and all the nodes form a cluster and this concludes this intro to kubernetes [Music] let's see how you can run kubernetes locally so do you need to install a kubernetes cluster in the cloud or ask your it Department to install one in your Enterprise so you can test locally absolutely not there are many ways that you can run kubernetes on a desktop or laptop Docker desktop lets you run kubernetes macrocates from the makers of Ubuntu and minicube also let you run kubernetes altree requires that virtualization is enabled kind runs over Docker desktop and offer extra functionalities Docker desktop is limited to OneNote but it's usually not a problem Marco Cates kind and minicube can emulate multiple worker nodes on Windows Docker desktop lets you run both Linux and windows containers and you can't create and run Windows containers on Mac and Linux it runs on hyper-v or Windows subsystem for Linux so if you have Windows 10 version 2004 or later it's the recommended way to run Docker desktop if fiber-v is enabled on your laptop or desktop you can't run another hypervisor at the same time and mini Cube used by default virtual box but it can also run on hyper-v you can install Docker desktop on Windows using App review and it will create a virtual machine named Docker desktop VM when you take the enable kubernetes checkbox in Docker desktop it will download additional containers to run kubernetes using Windows 10 version 2004 or later and if you have wsl2 installed you can tick the use the wsl2 base engine and Docker desktop will create its VM inside the Linux distro you install on your Windows machine note that this is the recommended way to run Docker desktop on Mac Docker desktop use the hyperkit lightweight hypervisor to run its VM mini cube is another popular option it does not require a Docker desktop it runs on Linux Mac in Windows and it requires an appervisor like a virtualbox here we can see mini Cube running on a Mac and its virtual machine in virtualbox if you need to install mini Cube on Windows but don't want to install virtualbox you can run minicube on hyper-v you need to create a network switch and start minicube with some extra parameters kind stands for kubernetes in Docker because it runs on top of Docker desktop kind lets you emulate multiple control planes and multiple worker nodes this is useful if you need to test node affinity and this concludes the sector on how to run kubernetes locally [Music] and this will be a super quick lab just to validate that our kubernetes installation is working locally so here I'm on Windows and I'm using Docker desktop and I installed kubernetes with Docker desktop in the system tray I can right click on the docker desktop icon on a Mac you can do that from the top of the the screen and I will select dashboard and from there we're going to click here on the gear icon the settings icon and I'm using wsl2 to run Docker desktop so that's the recommended way and here if I click on kubernetes uh enable kubernetes is checks so kubernetes is installed but is it running correctly let's find out let me go back here let's open a terminal and let's run uh this command uh Cube CTL cluster info that should give us a little information about what's running so kubernetes Master it as in green yay it's working so it's running at this address and Cube DNS is running at this address so by running qctl cluster info you get some information about the kubernetes installation and itself [Music] let's see how you can use the kubernetes CLI the kubernetes API server is a service running on the master node it exposes a rest API that is the only point of communication for kubernetes clusters you define the desired state in yaml files and let's say you want to run a x number of instances of a container in the cluster using the kubernetes CLI you then send that desired state to the cluster via the rest API other applications like a web dashboard can also communicate with the rest API to display the cluster state Cube CTL is the kubernetes CLI and it runs on Mac Linux and windows and you pick your choice of pronunciation cue control Cube cuddle Cube CDL doesn't matter it communicates with the API server and its connection information is stored in a config file under the dot Cube folder let's now see what a context is it's a group of access parameters that let you connect to a kubernetes cluster it contains the cluster name a user and a namespace the current context is the cluster that kubernetes commands will run against let's say that you can connect to three clusters cluster a cluster B and cluster C C C when you set the default context to Cluster B then all the cube CTL commands that you you will run will run against cluster B here's a cheat sheet for context commands Cube CDL config current context will get you the current context get context will list all of them Cube CL config use contacts and the context name Will Set the current contacts and delete contacts with the context name we'll delete the context from the config file there's a large ecosystem of free open source kubernetes tools that you can use Cube CTX is a good example it's a shortcut for the Cube's DL config use context command you simply type Cube CTX followed by the context name to quickly switch contacts it runs on Windows Mac and Linux very useful and this concludes the structure on the kubernetes CLI and the concept of context [Music] a context contains connection formation to a kubernetes cluster and you can have one or more than one context set on your machine it's super important to know how to figure out in which context you're currently in and how to change context so first thing first let's figure out in what context we're currently in I'm going to use cubectl config current context and this will print the name of the context we're currently in I'm currently in the docker desktop contact means that whenever I type Cube CTL commands it'll be applied to that kubernetes cluster all right I can have more than one context configure on my machine or on any machine or server to list them we use Cube CTL config get Dash context and here we can see that I have two contacts set on my machine Docker desktop which is the current context because it has that star in the current column and I have a second one called demo then one is a cluster that I created in the cloud what if I want to change from Docker desktop to demo okay let's use cubectl config use Dash context and the context name demo and now if I print again the current context there it is demo so whenever I'll be typing Cube CTL command DZ yaml files this command will be sent to My Demo cluster somewhere in the cloud a cool tool is Cube CTX because it allows you to well instead of using Cube CDL config blah blah blah blah blah basically it's a shortcut if I simply type a cube CTX that will print the context that I've configured on my machine so demo is green means that it's the current context and I can quickly change context using cubectx and the context name Docker desktop you know a little bit less keystroke to talk about anyway it's a fun too foreign context let's say uh you you your cluster has a long funky name and you want to rename it to make uh more sense you can use the cube CTL config rename contacts all name new name so let's try to rename our demo cluster here demo let's say it's on Azure so Azure demo okay let's use the cube CTX to print the contacts and there it is azure demo where is that context information stored it's stored locally on your machine so I'm on Windows let's see where it is it's on the C drive users your username under dot Cube deduct Cube folder and there it is config so if I right click on it and select opening with code we can see that it's a yaml file and it contains two entries here two clusters one is the Azure one and the second one is the Docker desktop one and this is the different context that I have that have so here is my Docker desktop context and here is my demo context well the cluster name is still demo but I renamed the context name to Azure Dash demo there it is all right let's go back here and let's say that I've deleted my cluster in the cloud I don't need it anymore I want to get rid of the contacts information I can use cubectl config delete contacts and the context name so let's try this Azure demo deleted context Azure demo from the config file so let's take a look here and you see that now I have only one contacts and here's the current context in this line here but let take a look here at the Clusters list My Demo cluster is still there right so it's not deleted automatically what you can do is simply edit that config file and remove the section no longer needed [Music] let's talk about the declarative and the imperative ways to create resources in kubernetes there are two ways that you can use when you want to create resources in kubernetes the declarative way and the imperative way way way using the imperative way you use cubectl to issue a series of command to create resources in the cluster this is great for learning testing and troubleshooting using the declarative way you define the resources needed in yaml files and use cubectl to send the content of these file as the desired state to the cluster so instead of a series of command this is reproducible and you can even store these yaml files in a source control system here we can see a series of command to create resources that's the imperative way way way you can create a pod using the Run command create a deployment or service using the create command using the declarative way you would use a yaml file to define the resource and then send the content of that file to the cluster to create these resources so what's a yaml file well it's a text file that contains properties that Define the resource it has some required properties like the API version the object kind that defines the type of object you want to create we'll take a look at these later on you can use the cube CTL create command to send the information to the kubernetes cluster we will take a deeper look at yaml files in the future lectures but right now you may be wondering do you need to type all that yaml manually the answer of course is no one way to get the correct syntax is to copy one from the official kubernetes documentation at kubernetes at IO slash Docs you then search for the object you want to create and click on the copy icon another way is to use templates offered with an editor like Visual Studio code let's say you create a new yaml file then you type control space and select the template to generate the Manifest that you can edit neat you can also use the kubernetes CLI to generate the ammo add dash dash dry run equals client and dash o for output specifying the yaml to Output the yaml to the console you can even send the output to a file using the greater than sign and a file name and this concludes this overview of the imperative and declarative ways to create resources in kubernetes [Music] let's deploy an nginx container using both the imperative way and the declarative way using the imperative way uh we're gonna type command called cubectl create I'm going to create a deployment I'm going to name our deployment my nginx1 I'm going to specify a parameter called image where we will specify the image we want to run this will create a deployment and we can Cube CTL get a list of the deployments using Cube CTL get deploy and there it is now the second way is the declarative way way way instead of typing a command with all the parameters at the common line we're going to specify a yaml file where all the configuration options are stored cubectl create Dash f for file and the name of the file so let's run this okay it was created let's again type Cube CTL get deploy get both our deployments and we if we take a look at the yaml file well it's a yaml file the type deployment it has the my nginx2 name it has a bunch of parameters that will come back to that a little bit later but basically all the configuration parameters are stored in that yaml file what's cool with that concept is that it's you can put these files quite easily in a source control system all right uh let's do a little bit of a cleanup let's delete our deployment Cube CTL delete deployment my nginx one okay and I'm gonna use a the same command but using a shortcut this time Cube CTL delete instead of deployment and the name of our deployment okay and if I type again Cube CTL get deploy there's no deployments currently my namespace [Music] let's take a look at namespaces so what is a namespace it's a kubernetes resource that allow you to group other resources let's say you need to deploy your application in multiple environments like Dev test and prod well you can create a namespace for each of these environments they are like logical folders in which you group resources kubernetes creates the default namespace called well default objects in one namespace can access objects in different in a different one the kubernetes internal DNS assigned Network names to some resources deleting a namespace will delete all these child objects this is super useful when doing tests create a namespace in the morning create resources under that namespace during the day and at the end of the day simply delete that namespace this command lists all the namespaces existing in a cluster we'll take a look at this command in more detail okay first you create a namespace this yaml file Define a namespace called prod then you use that namespace when you create other resources in the metadata section you set the namespace key to the name of the namespace you want this resource to be created in so namespace colon prod you can assign Network policies and limit the resources that you can create in a namespace using the resource quota object here's a cheat sheet for the namespace commands so cubesatel get namespace list all the namespaces and if you don't want to type namespace each time you can use a shortcut an S so Cube CTL get an S is the same thing as cubectl get namespace you can set the current context to use a namespace in the next commands that you'll type by using cubectl config set contacts current with which will use the current context and then the namespace is called the name space that you want to use so the next command that you'll type will be under that namespace cubectl create NS and the name of the namespace so you create a namespace you delete it using cubectl delete in s and the namespace name and you can also list all the pods or the objects from another namespace so Cube CTL get parts or any objects and then you pass the flag dash dash all- namespaces that will list all the objects in all the different things spaces and this concludes this look at namespaces [Music] let's see how to list and switch between name spaces you can get a list of the namespaces using the cube CTL get namespaces command let me copy this and these are the namespaces currently created on my machine on my kubernetes cluster so default the cube node Cube public Cube system I can also use the shortcut for namespaces which is just few letters which is a lot faster to type and it will give you the exact same result here let me get a list of the pods running Cube CTL get parts no resources found in default namespace that I'm currently my contacts is using the namespace called default right and there's no pods running in there so what if I want to list the pods that are in the cube system namespace well I can use the same command qctl get pass specifying the namespace switch and specifying the name of the namespace let me copy this and there are many parts running here here I can use also the the shortcut so instead of typing dash dash namespace equal I can just use the shortcut dash n so only one dash here two dashes here and the name of the namespace so let's try this and it works perfect now what if I want to change from the default namespace or the namespace called default to a namespace called Cube system right and then apply all my commands to so all the objects will be created in that namespace well I can do that by using the cube CDL config set contacts dash dash current so here we're modifying the context and passing a switch called namespace and the name of the namespace we want to change so we want to change instead of being the default namespace all the time we want to be in the cube system okay and yeah let's see if uh if we run Cube City I'll get passed if we would get some pods of course because we're currently in that Cube system namespace okay now let's change back to the default namespace and let's get a list of the parts there should be none perfect okay so this is how you uh you change namespaces perfect of course we can create new namespaces by using the cube CTL uh create NS for short for namespace and a namespace name so hello okay uh Cube CTL get NS and my name space here was created seven seconds ago and I can delete a namespace using cubectl delete NS and the namespace name that and hello now a bit of a warning here uh if you have resources under that namespace these resources the buzz the containers whatever running will also be deleted so use this command with caution foreign take a couple of seconds okay let's get the name spaces and our hello namespace is gone [Music] let's now look at the masternode in nowadays the physical or virtual machine a group of nodes formed a cluster the masternode is also called the control plane the kubernetes services and controller are located on the control plane they are also called The Master components and you usually don't run your application containers on the masternode the key value data Store where the state of the cluster is stored the API server is the only component that communicates with its CD let's start with the API server it exposes a res interface and client tools like the kubernetes CLI communicates through that recipe it saves the state in a CD and all client interacts with the API server never directly with the data store hcd is the data store for storing the cluster state it's not a database but a key Value Store is the single source of truths inside kubernetes the cube control manager is the controller of controllers its job is to run the other kubernetes controllers the cloud control manager job is to interact with the cloud providers a check if nodes were created or deleted uh Route traffic create or delete load balancers and interact with the cloud providers storage services the cube scheduler watches for paths that are not created yet and selects a node for them to run on it checks for various rules and then assign the Pod creation to a node finally you can install various add-ons on the masternode these add-ons provide additional functionalities in your kubernetes cluster and this concludes this look at the masternode [Music] let's take a look at the worker nodes a node is a physical or visual machine a group of nodes forms a cluster there's a special node called the masternode sometimes called the control plane where the kubernetes services are installed the nodes running the containers are called the worker nodes when a worker node is added to the cluster some kubernetes services are installed automatically the container runtime the cubelet and the Q proxy these are Services necessary to run pods and they are managed by the master components on the masternode the cubelet manages the pod's life cycle and ensure that the containers described in the Pod specifications are running and are healthy the Q proxy is a network proxy that manages Network rules on nodes all Network traffic go through the cube proxy on each node you will find a container runtime kubernetes supports several container runtimes that implements the kubernetes container runtime interface specification or CRI one thing to note is that for kubernetes version previous to 1.19 the Mobi container runtime was installed and was receiving the container runtime interface call through a shim because it did not fully implemented the specifications this is not ideal as it added an extra step starting with kubernetes version 1.19 Mobi is no longer installed oh wait a minute Moby not installed that means that my Docker containers will no longer run if the docker container runtime is not installed right well the short answer is that your Docker images will run as is nothing to change its business as usual what change is that what you can do inside the cluster since the docker runtime is no longer installed you no longer can access the docker engine and issue Docker commands directly inside a node you'll have to use another tool called crease CTL but again that's if you SSH into a node and run commands directly on that note something that you don't do usually alright a node pool is a group of virtual machines all with the same size a cluster can have many node pools and each node pool can host virtual Machines of different sizes let's say that we have two node pools in our cluster the first one consists of VM without gpus and the second one with chip use remember that Docker desktop is limited to one node so basically you run the master components and all the application containers on the same node and this concludes this look at the worker nodes [Music] so let's get some information about our nodes I'm going to run a cube CTL get nodes and since I'm running on Docker desktop locally here I have only one node name is Docker desktop status is ready roll is master here's the revision the version number and it was installed 72 days ago all right I can get more information about the uh the node uh since I only have one node I can skip the node name per meter and here's a bunch of information the name the row some labels creation date um what else what else what else what else uh the capacity to CPUs uh number of PODS maximum number of parts that I can run 110 the architecture the OS Linux architecture AMD the pods running and the CPU request CPU the Miss memory so all useful information when uh you need to troubleshoot and debug so that's pretty uh pretty interesting now I'm using a locally Docker desktop so I'm limited to one node Let's uh switch should to to to um My Demo cluster uh running in the cloud so let's do the same thing Cube CTL get notes I have more notes now so I have a cluster in in the cloud running three nodes in a node tool so one two three status ready agent and the version of kubernetes that is running that's interesting let me grab cubicity of the scrap node and let's uh let's say use this one so I can get the same information that I I was getting earlier but this one is running uh in the cloud that's the name role agent agent pool the different labels annotation number CPUs memory System Info it's running Ubuntu Linux AMD 64. uh what else will end the different parts that are running on this uh node [Music] let's take a look at parts so what are pots a pod is the smallest unit of work in kubernetes it encapsulate an application container and represent a unit of deployment paths can run one or more containers inside a pod containers shame the same IP address space and volumes and they communicate with each other using local laws that's inside the pod s are ephemeral deploying a pod is in atomic operation so it succeeds or not if a pod crashes it is replaced by a brand new one with a shiny new IP address you don't update the Pod that is currently running you create an updated version deleting the first one and deploying the new one you scale by adding more pods not more containers inside a pod I use this analogy in a previous lecture pods are like cattle there are ephemeras and you just replace them a note can run many pods and a part can run one or more containers if a pod runs multiple containers there's usually one that is the main worker where your application logic is located and the other ones are helper containers that provide services to the main worker we will come back to this concept in another lecture and this concludes this look at pods [Music] let's take a look at the Pod life cycle we'll start with the creation life cycle so when you issue a cube CTL create command to deploy a pod in your cluster the CLI sends the information to the API server and that information will be written into a CD the scheduler will wash for this type of the information look at the notes and find one where to schedule the Pod and write that information of course in a CD the cubelet running on the Node will watch for that information and issue a command to create an instance of the container inside a pod and finally the status will be written in a CD one thing to notice is that each time an operation is taking place inside the cluster the state is written in its CD so SCD is the single source of Truth in the cluster all right so let's take a look at the deletion life cycle now when you issue a cube City delete command to delete a pod from your cluster the CLI sends deformation of course to the API server that information will be written in SCD and notice that the grace period of 30 seconds will be will be added so the cubelet picks that information and sends a terminate signal to the container if the container hangs it is killed after the 32nd grace period and finally the state is stored in a CD The Path State will give you a high level summary of Where the Pod is in its life cycle pending mean that the Pod is scheduled for creation but is not created yet if you run out of resources in your cluster kubernetes may not be able to create new paths and if this happens the parts will be in the pending state running means that the part is currently running succeeded means that the code exited without any errors fail means that the code inside the the Pod exited with the non-zero status so some kind of error occurred unknown means that kubernetes can communicate with the pod and finally crash loop back off oh I love this in his state name so krashu backup means that the Pod started then crash kubernetes started it again and then the Pod crashed again so kubernetes say okay hold on I'm stopping here so we'll take a look uh in the official letter of where to look for these states and this concludes this look at the bud life cycles [Music] let's see how to define and run pods to define a pod the declarative way you create a yaml file specifying but as the kind that's the type of resource you want to create you specify an image location in this case the nginx image will be pulled from Docker up that's the default container registry you set the port that the container will listen on you can add labels they are used to identify describe and group related sets of objects and resources you can set environment variables directly here might not be the best idea to place configuration values in a later lecture we'll see how we can externalize that by the use of a config map object you can even specify a command to run when the container starts if you have created a yaml file with your pod definition you use cubectlcreate-f specifying the yaml file location and name and this will create your pod the declarative way now if you don't have a yellow file and you just want to run a pod the imperative way you use qctl run you specify a name for your running pod Dash Dash image the image name you can specify a program to run in this case it's sh and dash C you can specify a parameter that you want to pass to the program qctl get parts will list all the pods that are currently running Dash o wide will get you the same formation but uh with a few columns more cubicity Of The Scribe part and the Pod name will show the pad information you can use qctl get part with the Pod name Dash offer output uh in yaml format and you can pipe that to a file name so this is pretty cool because it will extract the Pod definition in yaml and save it to a file so if in case you lost that yaml file that was used to create a pod well you can recreate it quite easily qctl exec Dash it specifying the Pod name and the program to run will get you inside that part in the interactive mode you delete a pod using cubectl delete Dash F specifying the yaml file or if you don't have the yaml file simply use cubectl delete pod and the path name that will result in the same as the previous command and this concludes this look at how to define and run pods [Music] in this lab we will run our first pods we'll start by using the imperative way we'll use the cube CTL run command specifying the image that we want to run in our case we want to run an nginx web server and we'll specify a name to the running instance my nginx let's run this pod my nginx created let's get a list of the running pods by using the cube CTL get pods command I have one pod running my nginx ready one of ones that is running it was created 11 seconds ago let's try to get more information by adding the dash o y per meter it's the same command but we'll get a little bit more information like the IP address of the Pod and the node where the Pod is currently running awesome if you want more information we'll use the cube CTL describe command with the type of object and the name of the running instance so cubectl describe the type pod and the name my nginx all right tons of cool information here the name of the object my internet the namespace where it's running uh the node where it's running start time any labels annotation the IP address information about the container any volumes restarts that happen and any volumes here and at the end we get the list of events that happened when the Pod was created it was first scheduled and then the image was pulled and then the image was successfully pulled and created and then started the cube CDL describe command should be the first thing you try when doing some troubleshooting here you will likely have some very useful information uh in case a pod doesn't start maybe the image is not available maybe the image didn't start correctly also Cube CDL describe command that's very very useful let's now delete our pod by using the qctl delete the object type pod and the name of the Running part should get my command line back in a few seconds all right let's now run a second pod this time a busy box image Cube CTL run the image busy box and the name of the running instance my box but we're adding extra parameters here Dash it dash dash the program we want to run this will open a session inside our running pod and look at the command prompt it's changed to a pound sign this means that now I can type commands that are running inside the pod LS for listing the folders and files I can run a base64 command here I can encode a string here and that happened inside the running container inside the Pod pretty cool to stop the session simply type exit and the this ends the session okay let's do a little bit of cleanup um in the busy box case uh it takes up to 30 seconds to uh to delete so we have two options here we can run the cube CTL delete the object type and the running AdSense and to get back our Command Prompt right away we can use the dash dash weight equal false per meter or if we simply want to kill the Pod brutally we use dash dash force and with a grace period of zero seconds let's run this and this will kill the Pod brutally all right let's now create a pod the declarative Way by using a yaml file we have a yaml file here called myhab.yamo the kind is a pod we give it a name my app dashboard a few labels the image we want to run any limits here four degree sources in CPU and memory the port that the container is listening to and some environment variable we're defining an environment variable called dbcon that will have a value of my connection string here awesome let's use the qctl create Dash F4 file and the name of our yaml file pod my app.pod created perfect let's run the qcl get pods command to get a list of our paths it's running perfect we can also describe our pod again same information that we saw earlier right the default names the namespace and here look we have our environment variable and the same events that we saw earlier previously we used the cube CTL run command with the dash it parameter to open a session to our BusyBox container now what is the Pod is currently running well we can use the cube CTL exec command with the same it per meter here we specify the Pod name and the program we want to run that will open a session to a part that is currently running okay let's output the dbcon environment variable my connection string awesome that worked let's exit or stop our session and this time we'll use the qctl delete command specifying our yaml file to delete our pod [Music] let's take a look at init containers let's say that your app has a dependency on something can be a database an API or some config files you want to initialize or validate that these exist before launching the app but you don't want to clutter your main logic with this type of infrastructure code so what do you do you can use an init container that lets you initialize a pod before an application container runs let's say that for the app container to run it requires a series of configuration files in the Pod definition you define a container that will run first this is the init container upon completion kubernetes will start the app container this is a great pattern for applications that have dependencies the init container job can be as simple as validating that a service or a database is up and running this keeps the infrastructure code out of the main logic init containers always run to completion you can have more than one and each bus complete successfully before the next ones starts if it fails the cubelet repeatedly restarts it until it succeeds unless it's restart policy is set to never probe are not supported as ended containers run to completion in this bad definition file we have the main application located in the containers section in green and the init containers in the init container section in yellow as you can see here we have two init containers they both watch for services to be up and running so the first one we will run to completion then the second one and finally kubernetes will start the app container and this concludes this look at init containers [Music] in this lab we will use an init container to modify the home page of an nginx container let's take a look at our yaml file it's manifest for a pod um we have two sections here containers and any containers let's first take a look at init containers we will use a BusyBox image and we'll run this command wget and will it that website right here and that website it's the ohm of the first website the home page of the first website is pretty simple just a few lines of HTML all right and we will save that HTML into a file called index.html into a volume called work directory and the nginx image will map that value and we'll serve that index.html page as its default web page so basically we're initializing our nginx container by using an init container here awesome let's open a terminal and let's deploy our application Cube CTL apply Dash F the name of our yellow file let's wait till the nginx image is up so if I do a doctor PS yet my nginx container is a oh let's uh open a session right into that nginx container and let's try to hit the default web page curl localhost and yep that worked we're serving the default uh web page of the CERN website here pretty cool let's type exit and let's do our cleanup cubesatel delete and our yellow file [Music] let's now look at selectors when defining kubernetes resources you can use labels these labels allow you to identify describe and group related set of objects or resources they are simply key value pairs that you define yourself in this part definition we see two labels app app app with the with a value of my app and type with a value of front end note that the app and type keys are not something defined by kubernetes they are defined by you for your application okay but what does labels have to do with selectors well selectors use labels to filter or select objects here we see a selector in this pod definition the selector type is node selector and the key value pair is this type equals super fast okay but how does that work here we have a path definition with the node selector set to this type equals super fast we're telling kubernetes that we want to run this bar on the Node that has a label set to this type equals super fast note a does have such label so kubernetes will schedule the Pod creation on that note the simplest way to picture what selectors do is by comparing them with a SQL query it would be something like select star from nodes where this type equals super fast and this concludes this look at the selectors concept [Music] in this lab we will test the selector concept we have two yaml files here the first one one one it contains the definition of a pod we will run an nginx web server listening on Port 80.

and for the sector concept what is enter is is this section the label section so we have two labels defined here app set to my app type set to front dash n let's take a look at the second yaml file that's the service of name my service it's listening on Port 80 and targeting or redirecting the traffic to Port 80 that the Pod is listing on and let's look at this section selector app my app type front dash n so these are the same labels that we Define right here in our pod definition all right Let's test this concept I'll open a terminal and it will first deploy r by using cubesatel apply Dash F and the name of the yaml file foreign let's now deploy the service Cube CDL apply Dash f myservice.yaml all right how do we know that the service is connected to the Pod how do we know that the selection was successfully made to find that let's get the Pod IP address by using qcdl get pod Dash o wide and let's look at the IP address here 10.1.9.31 all right let's run this command cubectl get EP EB is short for endpoint and here's the name of our service my service so let's run this all right you see D in the endpoints column 10.1.9.31 that's the same IP address let's now try to port forward to that service okay we get an immediate result here and let's uh go to localhost port 8080 and that worked perfect let's go back here to our terminal and let's type Ctrl C to stop the port forward now let's try to break things a little bit bit bit we'll open the my app.yaml file and let's change the one of the labels so uh app will set it to my app to control s let's save that file and let's redeploy the file again the Pod again by using cubectl apply Dash F and the name of the yaml file all right let's check the endpoint again is it still working huh huh huh look at the endpoint colon none and if we try to pour forward again there's no immediate results so that doesn't work so disproved that both labels must match here the selector here boat labels must match the labels in in the Pod definition so that the selection can work all right let's type Ctrl C to stop this and let's do our little cleanup by deleting the service and then the pod thank you let's take a look at multi-container pods we sign a previous lecture that pass can run one or more containers that there's always a main worker and that the other containers are providing services to that main worker like saving data to a database and writing log files there are scenarios while multi-container Parts make sense and they are well documented in a series of patterns we will take a look at some of them in the next few slides with these sidecar pattern the helper container provides extra functionalities to the main worker let's say that our app writes some log files inside the pod the sidecar can copy these log files to some purchasing storage offered by the cloud provider this way the application code located inside the main worker is not cluttered with infrastructure code that code is located in the helper container and if you move from one cloud provider to another one well you simply replace or update that L per code this keeps your application code super clean our next pattern is the adapter let's say that our main worker outputs some complex monitoring information that the monitoring surface of our cloud provider cannot understand the adapter role would be to connect to the main worker simplify the data for the monitoring service again the code specific to the cloud provider service is located inside the helper container the Ambassador pattern is another type of the men of the middle role let's say that our application code needs to write to some nosql database but the code has no clue on how to do that no problem you can send that data to the Ambassador that in turn will write the data to the nosql data store the code specific to the data store service is located inside the helper container if you're curious about these patterns I suggest you get a copy of the design distributed systems book the other is Brennan Burns Brendan worked at the Google where he co-created kubernetes he now works at Microsoft so how can you define multi-container paths well if you remember the lecture about yellow files you see that the container sections is actually a list this means that you can Define multiple containers are added in yellow you see container number one and in green container number two and when you create a pod both containers will be created at the same time pretty cool here's a quick cheat sheet for multi-container parts so after you created your yaml file you simply use cubectlcreate-f specifying the yaml file name so same thing as creating a single container pod if you want to exec into one of the container you simply use Cube CTL exec Dash it the part name and specifying Dash C and the container name this way you can jump into one of the containers running inside the pod you get the logs for a container using cubectl logs the Pod name Dash C the container name and this concludes this look at multi-container pods let's take a look at some networking Concepts kubernetes is what we call a flat Network as a result most resources inside a cluster can see each other all containers within a pod can communicate with each other all pods can communicate with each other all nodes can communicate with all pods and all nodes paws are giving an ephemeral IP address while services are giving a persistent IP so this is quite important we'll come back to that later let's illustrate the cluster Network in blue here each pod gets an IP address and the containers inside a pod share the same address space the containers inside the same pod share the same IP address but each must be assigned a different port number they can communicate inside the Pod using localhost and the port number they can also share the shared volumes what about communication between pods can the container on the right talk to The Container inside the pod on the left using localhost no they can't they need to go through a service that will front the network traffic for the pod for external access to the cluster traffic goes through a load balancer service offered by the cloud provider in future lectures we'll look at different type of services that we can use in kubernetes and this concludes this quick look at networking Concepts thank you let's create a multi-container pod using a yaml file let's take a look at the yaml file so the kind is part the name will be two dash containers and in the container section we're defining two containers so the first one we'll use the nginx image and we'll name it my nginx and that web server will listen on Port 80. and we're defining a second container using the BusyBox image and we'll name it my box this one will listen on Port 81. and for the BusyBox container to stay up and running we need to tell it to to stay up by issuing a sleep command so it'll stay for an hour what we'll try to achieve is open a session on the BusyBox container and try to hit the nginx container so the the default web page served by the nginx container all right let's try to create our pod here using a cube CTL create and the name of our yaml file okay pad two dash containers created so let's try to see uh if they're running all right two out of two because we have two uh containers running in that pod status running 10 seconds uh the IP address so there's one IP address assigned to the to the pub and denote okay let's try to get a little bit more information by using the cube Ctrl describe pod and the name of the pod let's scroll up a little bit so the name is two containers then space is the default one the node where it's running uh the containers so the first one is my nginx which using a the nginx image we have information about the limits that we set earlier in the ml file my box using a BusyBox image listing on Port 81 right there's a that sleep command so useful information here and also look at the events now so earlier when we add only one container inside a pod we would get just one set of events for polling and creating the container now we have two first one is the Pod was scheduled and the second one here is on the second land line um the nginx image was in the polling state it was Paul created started and then the busy box container was pulled and created okay perfect now let's try to open a session inside our BusyBox container so we'll use Cube CTL exec Dash it okay here's the trick now we need to specify the Pod name and then the container name that we want to to connect to and then the the program that we want to run that's the trick when you have multiple containers you need to specify the Pod name and then the container name all right let's try to do that oh looks good at last yeah it worked okay now we'll use wget to try to hit that default page serve ID nginx web server wget with the this flag and we'll call localhost all right welcome to nginx so that word now Outpost for a couple of seconds and why I'm gonna ask you why did that work why calling localhost worked so that work because let's go back to our yaml file the nginx container is listening on Port 80. right we don't have to specify a import number here if the nginx container would have listened to something different we would have to specify Here Local O's colon and the port number all right let's try to exit this perfect let's now delete our pod using Cube CTL delete and the name of the ml file and by using the force and grease grace period equals zero flag that will kill both containers immediately perfect [Music] this is a super short lecture just to introduce you to the concept of a workload so a workload is an application running on kubernetes all containers running in a kubernetes cluster must run in a workload the Pod is the atomic workload it represents a set of running containers and all workloads will create pods the replica set and the deployment will provide extra functionalities on top of the pod like the ability to Define how many instances of a part we want the stateful set and the Daemon set are specialized workloads and finally the job and crunch up offer tasks that run to completion these are short-lived tasks we will see each of these workloads in detail in the following lectures and this concludes this super short lecture on the workload concept [Music] let's take a look at replica set the replica set primary job is to manage d-pad replicas making sure that the desired number of instances are running this provides the self-e-link capabilities in kubernetes while you can create replica set the recommended workload is the deployment now welcome back to that in a moment so let's say that you want three instances of a pod to run at all time you create a replica set and specify that you want three replicas if for some reason one bird crashes kubernetes will replace it automatically without any human intervention I'm pretty cool eh let's see how to define a Rebecca set by starting with a pod we take the Pod definition except for the API version and kind and we place these values in the template section of the replica set the final result is a replica set yaml file so basically in the section highlighted in green you will find values specific to replica set and under the template section the values that Define the Pod you want to run run run here you set the desired number of instances with the replicas property again while you can create replica sets the recommended workload to use is the deployment because it provides extra functionalities on top of the replica set set set so why bother and learn about replica sets well in the deployment lecture you'll see that when you create a deployment that will also create a replica set in the background that's why it's important to learn about the replica sets functionalities here's a cheat sheet for replicasets command you create one using cubectl apply Dash F and the yabl file you get a list of the replicases by using cubectl get RS you get some information about the replica set by using Cube CTL describe RS and the replica set name you delete one using Cube CTL delete if you have the yaml file you specify Dash F and the yaml file name or if you don't have the yaml file simply by using the replica set name using cubectl delete RS and the replica set name and this concludes this look at Olympic assets [Music] let's create three instance of energy next container using the replica set template so let's take a look at our yaml file the type of object we want to create is the replica set this will be the name of the replica set rs-example we want three replicas running uh at the same time and uh we want three replicas of this containers uh name will be nginx and the image nginx uh colon Alpine so it's a smaller version and we Define the resources and also the ports that each is listening on all right let's try to uh create that so Cube CTL apply or create Dash F and the name of our EML file okay we'll pick a set created let's take a look at our running pods and have three pods running okay and look at the names that were um that were assigned to the each of these spots RS Dash example and then is some kind of a unique number here each one must have a different or unique name so each one is ready running and look here each one has a different IP address perfect so let's take a look at our replica sets that we've created so Cube CDL get RS so there's one rs-example um three desired three current three ready everything is green everything's okay Let's uh now describe our replica set so Cube CTL describe RS for replica set and the name of our replica set so let me paste that right let's scroll up a little bit so the name is RS example uh it's running in the default namespace um any labels or annotations are listed here number of replicas that we set so three current three desired and the the current part the status S3 running and zero failed to The Container information so nginx we want to run the nginx Alpine image the sync on Port 80 the limits and so on and this these are the events that were raised here so each pod was successfully created here all right so last thing we need to delete what created using qctl delete Dash F and the name of the EML file thank you let's take a look at deployments we'll start by comparing pods and deployments Buzz don't self-heal meaning that if a pod dies kubernetes will not replace it automatically using a pod definition only one instance can be created and you can't update and roll back pods deployment can a deployment manages a single part template so you create one for each microservice you want to run when creating a deployment this will also create a replica set in the background but while you can see it you don't interact with the replica set directly you let the deployment manage that for you to summarize the replica set provides the self-ealing and scaling functionalities while the deployment provides updates and rollback functionalities let's take a look at the deployment definition you define the desired number of instances with the replicas property this will be used by the underlying replica set you set the number of iteration you need to keep using the revision history limit property and you set the update strategy in the strategy section you can set the strategy type to a rolling update this way kubernetes will cycle when updating the pods the other strategy is recreate kubernetes will take all existing pods down before creating the new ones we'll have a dedicated lecture on this topic later on like a replica set we start with a pod definition and we start the metadata section in the template section of the deployment the final deployment definition looks like this highlighted in green we see the properties specific to the deployment and in yellow the ones defining the Pod we want to run here's a cheat sheet for deployments command so if you don't want to use a yaml file you can create a deployment using the imperative way so you use Cube CTL create deploy you specify a name then with the image property you you specify the image replicas the number of replicas you want to run and you can specify also other properties like the port number that the paths will listen on if you have a yaml file well you simply use Cube CTL apply Dash F and the yaml file you get a list of the deployments using cubectl get deploy and you get the deployment info using cubectl describe deploy and the deployment name since a deployment will create also a replica set you can get the list of the replica set using cubectl get RS and you delete a deployment by using a EML file so Cube CTL delete Dash F the name of the yaml file or if you don't have it simply use Cube CTL delete deploy and the deployment name and this concludes this look at deployments thank you [Music] let's use the deployment template to create three instances of an nginx container so let's take a look at the EML file this time we want to use the deployment Kai Kai Kai type of object we want to create is the deployment we name it deploy example we want three instances three replicas and we want to keep three versions three replica sets versions in the history in site kubernetes I know if you scroll down a little bit we see that we want to run an nginx uh the Alpine version because it's a little bit smaller and we name it nginx we set some resource limits and each pod will listen on Port 80.

all right quite similar to the replica set template that we saw earlier except maybe for uh this parameter here all right now let's create our deployment using cubesatel apply or create Dash F deploy example and let's get a list of the pods that are currently running Cube CTL getbots Dash o wide and yes we have three parts three lines here so look at the names uh given to each bot so deploy example that's the name of the deployment object the the name of the deployment object Dash something unique so to make sure that each part has a unique thing kubernetes as a random number like this only one container is running inside each bot uh it's in running State each pod gets its own IP address so that's perfect okay let's now try to describe our pods so we can use the cube CDL describe pod and deploy example name but we just saw that the the names work a little bit different let's see if this works yep it worked it worked because uh the name is share across uh these three parts and so we get information about each one of these let's say I just need the information about that particular part I can use that name uh the unique name also so instead of deploy example I'm gonna use the full name and there it is f uh the information about that pod just that button all right let's now get some information about the deployment currently uh inside my cluster so cubesatel get deploy I have only one deploy Dash example three out of three ready up to date available so everything's uh looking good and I can describe my deployment using cubectl describe the name of the the type of the object and then its name all right so let's go a little bit we have the name the namespace where it's running any labels annotations number of replicas three desired three updated three total three available perfect strategies rolling update okay because we haven't set a running update strategy that's the default one we'll come back to that uh later on the path template nginx Alpine listening on Port 80. right and the events here all right since a deployment will automatically create a replica set let's take a look at the replica set that was created so Cube CTL get RS and for sure we have one replica set that was created by the deployment if we desired three current we're ready everything looks okay and we can also describe our Republic asset since we have only one we can uh use Cube CTL describe RS or we can use its full name we copy that and here I have the replica set description and finally we need to do a little bit of cleanup we delete our deployment using Cube CTL delete and with the name of the yaml file thank you let's take a look at demon sits the demon set is a specialized workload its role is to ensure that an instance of a pod is running on all nodes the paths are scheduled by the scheduler controller and run by the demon controller as nodes are added to the cluster the pods are added to them typical views are running some kind of helper service like a log collection or some kind of monitoring well let's illustrate that in this cluster we have two nodes and the demon workload ensures that an instance of a pod is running on each one of these nodes so you define a demon set in a yellow file you can specify that you don't want to schedule a pod on the master node by using a toleration same thing if you want to run the part on specific node pools here's a cheat sheet for the demon sets command so you create a demon set using a yaml file with the cube CTL apply Dash F command you get a list of demon set using Cube CDL get DS you get some information about the running demon set using cubectl describe DS and the name of the demon set and when you want to delete a demon set you either use a yaml file using cubectl delete Dash F and the name of the yaml file or the name of the running Daemon set by using Cube CTL delete DS and the demon set name and this conclude this look at demon sets [Music] let's run a BusyBox image as a demon set to ensure that we have one instance of that container running on each node in our cluster so let's take a look at the yaml file all right the kind is demon set we give it a name demon set Dash example and the container that we'll run is a busy box and here in the Toleration section we specify that we don't want to schedule a demon set on a node role that is mastered so we don't want to to run that demon said that the Busy box uh on the control plane on the masternode all right Let's uh deploy our demon sets so here I'm I'll be deploying that demand set on a cluster in the cloud that has three nodes because if I would try to do that on Docker desktop with just one note won't be interesting so let's try to do that doesn't matter what cloud provider I'm using right now okay uh let's uh get a a cube City I'll get parts we'll come back to uh the rest just after that so here I have three uh Parts demon example with a unique name here let's uh add the dash oh wide a flag Let's uh examine the node column so you can see that each pod is running on a unique node so 0 3 and 4 here and this selector is interesting because it if you have multiple objects you can basically filter out or just select what you want so here selector says the app uh equals the demon set example that's the name of our object here so that word one instance of our busy box is running on each node let me add a fourth node all right so my note count is now four let's go back to visual studio code and let's rerun uh the get paths command and for sure I have a Ford instance of my BusyBox container running as a demon set on the new note without me doing anything it was deployed automatically because I selected the demon set object type now we simply need to delete that demon set using a cube CTL delete [Music] let's take a look at stateful sets let's say that you run a database inside your kubernetes cluster traffic gets higher and you need to scale that database so you create more instances the main one can read and write while the other instances are in read-only mode and use replicas of the main database this is a complex scenario to implement the stateful set role is to help you solve this complex problem for parts that need to maintain state and unlike a deployment a stateful set maintains a sticky identity with a persistent identifier for each of the pods and if a part dies it will be replaced by another one using the same identifier the stateful set will create the parts in sequence and delete them in reverse order typical use are for Network related services that maintain some State and also for databases each pod gets a unique identifier using the stateful set name plus the sequence number and if one dies it is replaced by a new one but with the same identifier the pot creation is done in a ordered way meaning that the first one will be created then the second one and so on and when they are deleted they are deleted in the reverse order note that the persistent volumes are not automatically deleted foreign set you need to use a needless service you define one by setting the cluster IP value to none the stateful set will then refer to the atlas service and you define cloud storage in the volume claim templates section let's represent this slide as you can see only the first spot lets you read and write so how can you reach it if you want to write to the database well you use the pad network name in this case the instance name my SQL Dash zero plus the service name so dot MySQL when reading simply use the service name this will load balance the calls across all instances a bit of warning here containers are stateless by default and stateful sets are offering a solution for a stateful scenario but a lot of work has to be done on top of that a better approach might be to use the cloud provider database Services instead of trying to fit a stateful scenario inside your kubernetes cluster lastly deleting a sitful set will not delete the PVCs you have to do this manually here's a cheat sheet for stateful set commands so you create one using the cube CTL apply Dash F and the yaml file you get a list of set full sets using cubectl get STS you describe one using cubectl describe SCS and the set for set name and you delete a simple set using the delete command either using a yaml file or the name of the stateful set and this concludes this look at stateful sets [Music] let's now create a stateful set let's take a look at the state we'll set that EML file there are two parts in this CML file the first part is the creation of the Headless service the the kindest service but the trick here is to set cluster IP To None like this this will create an endless service all right second part is to create our stateful set so the kind set will set give it a name nginx Dash SCS and here we're referencing our Atlas service here so the name of our Atlas service assigned to the service name property we want three replicas right of a nginx image and we're creating a claim on a search class name called Azure file so the cloud provider here is not important that will work on any cloud provider so I already have a search class name called Azure file we'll use the read write once access mode and we'll use when when gigabyte of storage and we'll Mount this to a folder in VAR www all right so let's create our Seattle set here Cube CTL apply Dash if stateful set Dot yaml and quickly let's do a cube CTL get but a white and here you can see that the first instance is running the second one is running the third one is pending so each instance will be created in a sequence so the first one zero the second one one the third one two you can see by the age of each instances so 19 seconds 14 and 9 seconds they are created in a sequence and deleted in a reverse order also here we can see that um there's an IP address assigned to each of these let's take a look at the PVCs we should have a PVC for each of these instances and yes SCS 0 1 and 2. so uh we can do a mapping one to one here so for uh scs0 we have a PVC call also scs0 to prove that let's uh describe the second one let's say so Q the L describe paths and the number two and let's see what the PVC is assigned what claim is assigned so here the volume claim name scs-2 awesome okay what we'll do we'll create a file in the in that instance the instance number two and we'll delete the the Pod the Pod will be recreated automatically and we'll see if the file still exists we'll also modify the default web page served by nginx and see how we can reach that that file the default web page from another instance so let's open a session on the nginx CS2 perfect Let's uh CD in VAR are volume and let's uh simply type Echo and pipe that to a file called echo.txt I do it LS my file is there get Hello dot txt yeah perfect okay first step creating that file uh Second Step modifying the default web page let's CD into the user chair nginx HTML folder and let's do an LS here so here's the default page served by nginx index.html we will brutally replace that file by using cat and piping that to the file name and typing hello enter Ctrl D on Windows to save the file the file is there if I simply do a cat index.html yeah okay the file has been brutally replaced okay we'll close our session on this instance so scs2 let's close our ins our session and let's open a session on the instance zero okay awesome let's try to it that default HTML page but on the instance number two so nginx as CS2 so to do that we'll need to use the web the web address nginx scs2 so that's the name of the instance dot the name of the atlas service and that word would only the hard name word no you need a combination of boat the instance name and the atlas service also all right let's exit our session here okay let's try to delete uh the uh instance number two okay let's do a q that's CTL get but we have a new instance uh that was created uh seven seconds ago but the uh the name of the instance is still the same so instead of a random number here by using a stateful set this will ensure that the names will be the same okay so let's open a session on the instance number two let's LS far and here's our file awesome let's do our cleanup so we'll delete the stateful set and we need to manually delete dpvcs because simply deleting this table set will not delete the PVCs so let's do that [Music] let's take a look at jobs are for short-lived task workloads you start a job it executes and succeeds or fails so job don't say memory they don't wait for traffic a job creates one or more pod and ensures that a specific number of them successfully terminate the job tracks the successful part completions and then marks the job as complete when the desired number of completion is reached when using multiple Parts the job will create them one after the other they can also run in parallel this is a job definition you define how many paths you want to run at the same time you can set a deadline if needed and the number of completions to reach to Mark the job as complete and you should set the restart policy to never so here's a cheat sheet for the jobs command you create a job the imperative way using Cube CTL create job the job name and the image name using the declarative way with a yaml file you use cubectl apply Dash F into the yaml file name you list the jobs by using cubectl GitHub you get some information by using cubectl describe job and the job that is currently running and you delete jobs with either a yaml file using cubectl delete Dash F or with the job name Cube CTL delete job and the job name and this concludes this look at jobs thank you in this lab we'll create a simple job so let's take a look at the chub yaml file so the kind is chopped so that's the type of object we're creating uh the name we're giving it hello and what we will run is a busy box container and when the container starts it will Echo uh hello front the job perfect something sip simple Let's uh run this Cube CTL apply with the name of our yaml file perfect the job was created okay let's get a list of the jobs there's one hello completion one so it ran and duration was two seconds okay let's do a cube CDL describe job since we have only one job that will do the work if not we can type the job name hello and uh but but let's take a look at name hello namespace default annotation and so on and so on it ran One Time One succeeded so that's our container successfully create and complete it perfect now now now we we we can get a list of the pods using cubectl get that and this is uh the Pod that was created to run the job it's still there you see the status completed but it's still there so we can get to to examine the log in case something went wrong so we can do Cube CTL CTL CTL logs and the name of the part here hello from the job that worked so let's do our cleanup let's delete the job is deleted do we have any uh Parts left none any chops left none perfect [Music] let's take a look at crunch apps a crunch up is a workload that runs jobs on a schedule it's an extension of the job that we saw in the previous lecture the schedule is defined using a crown like syntax in UTC and you can get more information about the crown syntax in this Wikipedia page here's a crunch out definition you set the scheduled parameter to a cron schedule so how do you know if a crunch up ran successfully well you need to look at the job history by default the last three successful jobs and the last fill job are kept the paths will be in a stop State and you'll be able to look at their logs foreign if you don't want to keep any history you can set the successful job history limit to zero so here's a cheat sheet for the crown job commands you can create one using the imperative way and if you have a yellow file you use the cube CTL apply Dash F and the name of the yaml file you can get the list of decron jobs currently running using cubectl get the CJ you can get some information with the cube CDL describe CJ and you delete the crunch up using its CML file using Cube CTL delete Dash F and the yaml file name and if you don't have the yellow file name you can delete it using cubectl delete CJ and the crunch up that is currently running and this concludes this look at run jobs [Music] so let's create a crunch up we'll take a look at the yellow file kind is the crunch up I'm going to name it hello Cron and we will give it a schedule only Stars which means that it will run every 60 seconds every minute that's the default and we're going to run a busy box image and we it will Echo this string okay so let's create our job using cubectl apply and we can get a list of the crown jobs using cubectl get print jobs okay so here we have the name the schedule if it's suspended it's if it's active the last last time it ran and we can get some information using cubectl describe run job and we pass the its name hello Dash Cron here this is super useful for troubleshooting the name the default name space it runs on in the default namespace we have the schedule here how many jobs does it keep in its history how many failed job is kept also the the command that will run oh oh oh you can get a list of the pods okay uh one per Ran So elocron with a unique uh value here so it has uh completed I'm gonna pause um to let it run a few times all right uh the job ran three times well it the last one is still uh running uh container creating Let me refresh that so it's completed okay now we can get the logs by using cubectl logs and then the job name dpod name so hello from the crown chart so by default the last three run of the job are kept in the history and you can configure that in the yaml file let's delete our crunch up here perfect and if again we type Cube CTL get pads uh all the parts in in the history basically are also deleted when you delete the crunch up [Music] let's take a look at rolling updates in a previous lecture we saw that using deployments you can set the number of paths instances using the replicas parameter and set a number of previous iterations of the deployment to keep in kubernetes we also saw that there are two update strategies running a date and recreate recreate is quite simple kubernetes will shut down all the running paths and create new ones after that running update will cycle through updating parts all right let's illustrate that using recreate all previous paths are deleted and the New Path will be created after that this means that there might be a small period of time where your microservice might not be responsive using the routing of this strategy a pod is deleted and replaced by a new one one one then the next one and so on there are two values that you can set to help you with this process Max search will tell kubernetes how many parts can be created over the desired number of pods let's say that you you want three instances in total setting Max search to 1 will allow creation of one additional pod on top of these three desired ones and this while the rolling update is running Max unavailable is the opposite is the maximum number of pulse that can be unavailable during the update process note that if you don't specify an update strategy in the deployment manifest communities will use a default strategy of running update with Max Surge and Max unavailable both set to 25 percent so let's say that we want three instances of a bot and we set max search and Max unavailable to 1. we're telling kubernetes that it's okay to create one additional part on top of the three desired one and that's it's okay to have one part less than the three desired one when done the previous replica set is kept you set how many you want to keep with the revision history limit property here's a cheat sheet for running updates you create of course your deployment using cubectl apply Dash F and the name of your yaml file you get the progress of the update using cubectl rollout status you get the history of the deployment using cubectl rollout history deployment and the deployment name you can roll back a deployment using qctl rollout undo and the deployment name that will roll back to the previous version or if you want to roll back to a specific revision number you add the two dash revision parameter and this concludes this look at running updates [Music] in this lab we will create a deployment and later on update it to a new version using a rolling update so let's take a look at the yaml file the kind of object we're using easy deployment call Lo Dash dab we want three replicas and we're using a rolling update strategy here and setting a Max search to 1 and Max unavailable to one we will deploy a container called Lo Dash app yeah hello app and uh that'll be version one and later on we will update it to version 2.

okay so let's create our deployment here okay we can get a the deployment status by using Cube CTL rollout status and the deployment name uh the deployment was successfully rolled out let's take a look at our pods running okay so I have three instances perfect yellow depth uh three times excellent let's describe the uh the deployment so Cube CDL describe uh deploy and the deployment name and let's try to find if we can get some information about the strategy yeah here it is rolling update strategy Max unavailable oh wow the strategy type is right here uh rolling update running update strategy Max and available One Max search to one so you can get that information that we set earlier in the yml files okay let's now see if we have a replica set here's our replica set and let's Now update our yaml file and change the version of Hello app to version 2. so just update to 2 and save the file and we will use cubesatel apply and with the same yaml file now what I'm going to do how I'm using right now A K9s it's a terminal dashboard in a terminal sorry and to get the visual view of what's what will happen basically so right now I have three parts in green these are the ones that are deployed let's apply our new deployment I'm going to switch quickly to K9s and you can see oh it happens so fast but but but is you saw that the the new pods were were created and the old ones were uh were shut down all right here I can have a a deployment status everything's fine if not we would have uh some information if the deployment would take longer we would have some information printed here okay how many uh replica sets do we have we have two uh the the current one so that's the current one and the previous one by default uh three uh versions are kept uh in history we can get the deployment history by using cubectl rollout history and the deployment name okay version one for version two okay we're at version two and um we want to downgrade basically our roll back to the previous version so you can do that by using Cube CTL rollout undo and the deployment name so by default it will roll back to the previous version or if you have multiple versions and you know what version you want to roll back to you can specify the flag to revision and the revision number as we see here foreign let's do this one well either one will do the same thing okay let me switch to K9s and see what's happening oh the other one the the older while the version two is uh terminating new version is created and we can get a deployment status here everything successfully wrote okay Let's uh now take a look at our replica sets so we still have two right the current one is now the the first one that we uh that we deployed all right and we can do our cleanup by deleting the deployment using the yaml file [Music] let's take a look at blue green deployments so let's say that version 2 of our macro service contains some breaking changes like a different database schema what do you do using the rolling update strategy you'll have both version 1 and version 2 of your app running at the same time that might not work at all so using the blue green deployment pattern that might help solve that problem blue identify what's in production and green identify a new version currently deployed but not yet in production notice that the pulse label contains the version number when ready simply update the service definition to point to the new version and now green is now in production so green becomes blue and blue becomes green oh great this means that this pattern is solving the new database schema problem well not entirely you may still have to deal with some downtime while you update your database and also another drawback is that since both version of the macro services will be up and running at the same time you need to have enough free resources in your cluster to make this possible and this concludes this look at the blue green deployments pattern [Music] in this lab we'll create a blue green deployment we have three yaml files here so let's take a look at the lodep V1 it's a deployment we want three replicas of a container of a an image called Hello app 1.0 and here we're setting a label of app lov1 now let's take a look at the second one it's basically the same thing so deployment three replicas but it will use version 2 of our hello App application and we're setting the this label Here app to lov2 all right let's take a look at our cluster IP manifest file so it kind of service and here's the selector app hello V1 we will deploy that and later on we will change the cluster IP manifest to point to the newer version okay so let's deploy version one of our pods and let's also deploy our cluster IP service so let's take a look at the list of the paths currently running there are three pods and also there should be uh if I type Cube City I'll get SVC one cluster IP as we see front that's the one I just deployed perfect so let's do a quick port forward to uh to connect to our cluster IP service so we'll port forward uh the port 8080 that the cluster IP is listening to to localhost 8080. here cubesatel port forward service the name of our service 8080 28080 and let's hit local OS on port 8080 here okay so here hello world version 1.0 excellent day okay let's now deploy version two so uh right in my phenomenal I'll hit Ctrl C on my windows keyboard to break that and uh gain my terminal back perfect let's deploy version two okay and let's get a list of our pods currently running should be six so I have both versions uh in memory at the same time okay so that's one of the drawback of this uh this technique this blue green deployment technique okay let's now it did the cluster IP manifest file so we're we will change the selector to select app on Lo V2 V2 V2 let's save the file and let's update our cluster IP service by using cubectl apply and the name of the yaml file yes okay Let's uh port forward again right let's hit that local OS again and there you go V2 it worked let's do a little bit of her clean up uh let's delete our first deployment our second deployment and also decluster IP service you can select the three lines at the same time [Music] let's take a look at the concept of service in kubernetes first what problem do Services try to solve well if the pod in green need to reach the pod in purple you need to use its IP address the problem is that pass yet ephemeral IP addresses if the part in purple dies well you replace it and the new one will have a different IP address so we need a way to make these calls between pods a lot more robust so back to the service what exactly is the service well it's a kubernetes object that you define in a yaml file but on my pods that have ephemeral IP addresses Services get durable IP addresses and also they get DNS names they serve as ways to access Parts they're kind of a middleman and the target pods using selectors here we have four pods and a service the service select the pods that have the Zone label equals to prod and the version label equal to one the first part satisfies the selector the second one also but not the third one and not the last one so only the first two are selected so let's say we have two instances of a pod and we place a service in front of them if another part needs to reach these ones it will go through the surface and then the service will load balance the request to the instances in kubernetes we can use these Services the cluster IP the node port and the load balancer note that the cluster IP is the default service we will look at them in more details in the next lectures and this concludes this quick look at the concept of services in kubernetes [Music] let's take a look at the cluster IP so what is a cluster IP well it's the default service in kubernetes its visibility is cluster internal this means that it's not possible to use a cluster IP service to reach a macro service from the internet from outside the cluster in the cluster IP definition you can set two different ports Port is the port that the service will listen on and Target Port is the port that deselected pods are listening on so the cluster IP will route incoming traffic to that board in this CML file the service listens on Port 80 and Route the traffic to Port 8080. traffic is load balanced across the selected paths so when do you use a cluster IP service well to provide a durable way to communicate with pods but from inside the cluster so let's illustrate this here we have a cluster IP service fronting three instances of a pod it's impossible to reach it from outside the cluster but it's okay it's visible from inside the cluster this cluster IP will listen on Port ad and select the paths using these two labels since the selected pods are listening on port 8080 the service Target Port must also be set to 8080.

this way the parts in green that want to communicate with the ones in purple well they go through the cluster IP service on Port 80 and the service route the traffic to Port 8080. so let's say you have multiple Marco Services uh you place a cluster IP in front of each of them because a cluster IP service IP address is durable while the pods ones are ephemeral here's a cheat sheet for cluster IP so the first two commands are imperative commands let's say you already have a pod running and you want to expose it using a cluster ID so you would use Cube CTL expose Oh short for path the Pod name specifying the the port and the Target Port and you can also give a name to your to your service if you have a deployment you can also use Cube CTL expose deploy the deployment name specifying the port and the target board so both commands are imperative commands if you have a yaml file you would use Cube CTL apply Dash F and specifying the EML file name you can get a list of the services running using cubectl get SVC I get a little bit more information specifying uh the flag Dash o and wide you can also describe the service using a cube CTL describe SVC and the service name and you can delete the cluster IP service using the yaml file with the cube CTL delete Dash F and the name of the yaml file or cubectl delete SVC and the name of the service and this concludes this look at the clusterity [Music] in this lab we will deploy an nginx container front it with a cluster IP service and then deploy a BusyBox container open a session on that BusyBox container and try to hit the web page served by the nginx container but through the cluster IP service all right let's take a look at our yaml file we'll start with the deployment one kind is a deployment we want three instances of the Pod and we will run the nginx image the Alpine version and it will list it on Port 80. now we're setting two labels here and the cluster IP service will you will select uh the pods using these two labels so app example environment prod all right let's take a look at our cluster IP kind service it's gonna listen on port 8080 and it will redirect the traffic to Port 80 on the nginx containers all right the selector is here so app example environment broad so that will select our pods uh in our deployment perfect so let's try to do that first let's uh deploy the service and let's deploy the nginx containers let's also deploy the busy box and we can take a look at this it's a kind of pod the name is my box and uh it'll run a busy box image okay so now let's get a list of our Bots currently running we should add four one two three four so the first three ones are the deployment the nginx images and the fourth one is the BusyBox all right let's try to connect to the BusyBox container open a session by using cubectl exec my box Dash it and the name of program we want to run perfect at work by type LS yep okay let's try to use the service to reach the nginx pods so wget and HTTP SVC example colon 880 so let's try to run that and see if it works okay that worked why did it work so what is that name here if we go back to the cluster IP definition that's the name of our service and it's listening on Port 8080. the service name Colin d uh the port that is listening on and that's it we can now exit our session on our busy box and we can delete our resources the cluster IP the deployment and the pod thank you [Music] let's take a look at the note Port service what is a note port a noteboard extends the cluster IP service and provides extra functionalities its visibility is internal like a cluster IP but also external to the cluster you can set a third port using the note Port property this is the port that the service will listen on outside the cluster note that the port must be in a range between 30 000 and 32 767.

and if you don't specify a note Port value well kubernetes will assign one randomly you then set the port and the Target Port Properties like you do with a cluster IP this sounds like a good way to expose our macro services to external traffic but this range between 30 000 and 32 767 it's kind of annoying because you can't set it to let's say port 80. and One requirement for using note ports is that nodes must have public IP addresses to access your service simply specify any node Port IP address plus the note port and the traffic will be routed to the right note Port service inside the cluster the way it works is that you set the pods and the service just like you did before with the cluster IP but this time you also specify a port number in the note Port property external communication uses the node IP address and the port set with the node Port property internal communication uses the port set in the port property just like a cluster IP now let's take a look at our note Port cheat sheet if you already have a pod running in your cluster and you want to expose it using a note Port service simply use the cube CTL expose bow the Pod name specifying the port and the Target Port and the note Port as the type now you may wonder where do you specify the note port number well you can't there's no properties letting you set know that value between 30 000 and 32 767. so communities will assign one randomly for you same thing for a deployment let's say you have a deployment already running in your cluster and you want to expose it using a note Port you use cubectl expose deploy the deployment name Port Target Port type which is note port and you can specify your name also you can define a node port in a yaml file and deploy it using cubesatel apply Dash F the name of the yaml file you list the services using cubectl get SVC get more info adding Dash o wide he can describe your service using cubesatel describe SVC and the service name if you have a yaml file you can delete it using the that file using cubectl delete Dash F the name of the yaml file or you can delete your service using its name with qctl delete SVC and the service name and this concludes this look at the note Port service [Music] in this lab we will expose a deployment using a note Port service we have two yable files let's take a look at them the first one is for the deployment we will deploy an engine X image the Alpine version listening on Port 80. and we will need to replicas perfect let's take a look at the noteboard yaml file kind is service and the type is noteboard and the selector will select our deployment and here we set our node Port 32 410 okay now let's open a terminal and we'll start by the deployment cubesatel apply and our yaml file next our service Cube CTL apply noteboard.yaml awesome let's make sure that our pods are running Cube CTL getbots Dash o wide awesome two parts two instance of our nginx container all right now since we're using Docker desktop the docker desktop node is mapped to local OS to reach the note Port service we need to use local OS plus the note port let's try that local Post 32 410 and it worked awesome Now when using a cloud provider you would need to get a node IP address instead of using the Local Host you would get that IP address by using cubectl get nodes Dash o wide and here in the external IP address colon you would find the external IP address of the node awesome let's do our cleanup let's delete our note port and our deployment [Music] let's take a look at the concept of surfaces in kubernetes what problem do Services try to solve well if the pad in green need to reach the purple one it needs to use its IP address the problem is that pods are ephemeral if the purple one dies you need to replace it and the new one will have a different IP address we need a way to make these calls between pods more robust so what exactly is a service a service is a kubernetes object that you define in a yaml manifest unlike parts that have ephemeral IP addresses Services gets durable IP addresses and a DNS name they serve as a way to access paths and they target pods using selectors here we have four pods and a service the surface selects the paths that have the Zone label equals to prod and the version label equals to V1 the first part satisfies the selector the second one also but not the third one and the last one only the first two are selected let's say that we have two instances of a pod and we place a service in front of them if another part needs to reach these parts it will go through the service and the surface will load balance the requests to these instances in kubernetes we can use these services D cluster IP the note Port the load balancer and the Ingress the cluster IP is the default service its visibility is internal only the note Port can expose a pod outside the cluster the load balancer and the Ingress are similar Services they let you expose applications outside of the cluster one operates at D layer 4 and the other at layer 7.

L4 L7 what's that download balancer operates at the layer 4. that's the TCP transport level so that's very low in the transport stack it means that the load balancer can do simple operations like round robin routing the Ingress operates at the higher level in the transport stack think of protocols like HTTP or SMTP it's more intelligent so you can configure complex routing rules okay no worries if this sounds complex for now simply remember that an Ingress is like a load balancer but more intelligent and this concludes this look at the concept of services [Music] in this lab we will create a load balancer service but you may be may be asking yourself we're not using a cloud provider right now how can that work well Docker desktop is helping us it will emulate the load balancer service so we can test our load balancer locally awesome let's take a look at the application it's a simple deployment we want to replicas two instances of an nginx image super simple and the the load balancer yaml file kind is service the type is load balancer it will listen on port 8080 and redirect traffic to the Pod that is listening on Port 80. all right all right so let's open a terminal okay and let's deploy the app and the load balancer perfect let's make sure that our pod are running by using Cube CDL getbots yes I have two uh my coupons are here perfect now to get the IP address of the load balancer we use Cube CTL get SVC Dash o wide foreign S as the IP address using a cloud provider load balancer service you would find here instead of local OS the public IP address of the load balancer so let's test this open a browser and type localhost 8080 and that works we reach our nginx pod awesome let's do our cleanup let's delete our load balancer and our application [Music] this is an introduction to the persistent Concepts in kubernetes we saw this slide earlier containers are ephemeris and stateless and any data stored in them is deleted when the container is destroyed so we need to find a way to store data outside the containers if you want to keep that data so volumes let containers store data into external storage systems these are storage services offered by the cloud providers Defenders create plugins according to a specification called the container storage interface and there are two ways to use storage in the cloud the static way and the dynamic way we have separate lectures on these later on all right the cloud providers create plugins to expose their storage Services as persistent volumes and storage class these two are kubernetes objects next we will look at the static and dynamic ways [Music] let's see how to use the static way persistent volumes or PV and persistent volume claims are PVCs are two kubernetes objects that lets you define and use external storage a purchasing volume represents a storage resource that is available cluster wide and is provisioned by the cluster administrator you then use a persistent volume claim to claim the persistent volume a part will then use the PVC to mount a local folder PVCs can be used by multiple parts and inside the parts all the containers are sharing the same volume there are many persistent volumes plugins available some are offered by the cloud providers the one highlighted in yellow called hostpath is a special one it's a plugin available with kubernetes that allow you to do local testing and it's not mapped to a cloud provider storage service it will not work in a multi-node cluster but it's super useful for local testing here's the main drawback of persistent volumes let's say that the cluster admin provision 100 gigabytes of storage and that the Pod only requires a small portion of this storage let's say just one gigabyte of that 100 gigabytes in total so just one gigabyte well too bad for the other pods because the volume is used exclusively by the Pod who has the claim on it this can be a waste of precious resources and we will see how storage class get around this problem in the next lecture okay in the meantime let's focus on the PV and the PVC you first select the cloud provider storage service you want to use then you create a persistent volume and set the required capacity let's say here 10 gigabytes you then create a PVC so a claim that refers to the persistent value and finally you use the PVC from your pod and mount a local folder on it there's an inputs and property that you must be aware of it's the reclaim policy set a delete all data will be lost when the claim on the volume is released and this is the default value so be aware of this if you want to keep your files when the PVC is released you have to set the reclaim policy to retain again the default value is delete so be careful and be aware of this there are three access modes possible using read write mini the volume can be mounted as read write by many parts using read-only mini the volume can be mounted read-only by many parts and finally with read write once the volume can be mounted as real read write Sorry by one single path and the other parts will be in read-only mode this might be useful if you have a main worker that writes data and the other pods simply read the data you define a person in volumes using the purchasing volume kind and you specify the capacity the access mode and the reclaim policy in the spec section in this example the ospad plugin is used to access local storage remember to only use hostpat for local testing and refer to the storage provider documentation and on how to create a persistent volume specific for their storage service you then Define a claim so a persistent value claim making sure that the access mode match the one set in the processing value in this case the claim is for 8 gigabytes out of the possible 10 gigabytes set on the persistent volume this means that no one can claim the remaining two gigabytes until the claim is released in the volume section of your pod definition simply refer to the PVC and mount a local folder on it a persistent volume can have these states available meaning that the resource is free and not currently in use bound the volume is bound to a claim so it's in use it's not available anymore release the claim has been deleted but the resource is not yet reclaimed by the cluster and finally failed well something's wrong here's a cheat sheet for the PV and PVC commands using a yaml file you can create either a PV or PVC by using cubectl apply Dash F and the name of the yaml file you get the install persistent volume using cubecti Cube CTL getpv the claims using cubectl get PVC you can describe them cubectl describe PV or PVC with their name you can delete them using their yaml file cubicle delete F and the name of the yaml file or by using their name so Cube CTL delete PV and the PV name or the PVC name and this concludes this section as the static way next we'll take a look at the dynamic way [Music] in this lab we will create a persistent volume a persistent volume claim and use a pod to mount a local folder on that storage we will create that in Docker desktop locally using the host path plugin all right so let's first take a look at the persistent volume yaml file the kind is purchasing value we give it a name pv001 and a storage capacity 10 megabyte access mode read write once and we set the processing volume reclaim policy to retain and use that host Pat plugin here and to map to a folder in the docker desktop virtual machine to data here all right let's take a look at the PVC kind persistent volume claim we give it the name my claim read write access mode must be the same as the PV persistent volume so let's double check read write once on the PV read write once on the PVC awesome we request 10 megabytes of storage so we record the full capacity we could have chosen a lesser value if you want all right let's take a look at the Pod now so it's uh the kind of pod it's a busy box box box and uh we make a reference to the claim here in the volumes section we give it a name and we reference the persistent volume claim called my claim this guy here okay and we use that name mypd and we mount it to a local folder called demo we should see magically appear a folder called demo inside our BusyBox container okay let's deploy our persistent volume right percent in volume pv01 created awesome let's look at the PV qctl getpv name pv001 capacity 10 megabyte read write once reclaim policy to retain it's available it's not claimed uh awesome so let's now deploy the claim the PVC persistent volume claim my claim created awesome Cube CTL get PVC my claim it's bound to the volume called pv001 capacity 10 megabytes read write once and let's again take a look at the PV to see if something has changed yep it's now bound the status is bound to the claim called my claim running in the default name space awesome so let's now uh deploy our pod okay my PC box was created let's connect to it using Cube CTL exec the name of the instance so the Pod Dash it and the program you want to run so let's do NLS and see if we see a demo folder and there it is we have our demo folder Let's uh CD into that folder and let's create a a file inside cat and we'll pipe that to uh lo.txt let's type hello world world uh if I can type Ctrl D to exit and save the file Let's do an LS to see if the file was created perfect okay let's exit this session and now let's delete the pod let's delete the part by using cubectl delete Dash F pod and since the busy box takes uh 30 seconds to shut down we will force it to do it right away we don't want to wait okay Let's uh deploy it again well Cube CTL rctl get the pods no resource awesome it's really dead Let's uh deploy it again okay Let's uh open a session CD demo LS all right and let's get that file hello a car cat cat nut car hello world awesome it worked Let's uh exit our session let's now do our cleanup we'll delete our bud and then we will delete the PVC right and then the PV so you can't delete the PV before the PVC well you can issue the command but the the command will be in kind of a weight State uh until uh the PVC has been released [Music] let's continue our journey into persistence by looking at the dynamic way way way so here's a new object the storage class and the search class represent a storage resource that is available cluster wide and is provisioned by the cluster administrator you don't have to set a capacity and it eliminates the need for the admin to pre-provision a persistent value now compared with processing volumes where once a claim has been made the remaining capacity becomes unavailable well the storage class can support many claims many persistent volume claims so you first select the cloud provider storage service that you want to use you create a storage class so here no need to specify a capacity then you create a PVC the claim that refers to the storage class and now you specify the required capacity and finally you use the PVC in your pod and mount a local folder like a persistent volume there's an important property that you must be aware of it's the reclaim policy set at delete all data will be lost when the claim is released and it's the default value also like the persistent volume so be aware if you want to keep your files when the PVC is released you have to set the reclaim policy to retain again the default value is delete again three access modes possible and they are set using the PVC not the storage class read write mini the volume can be mounted as read write by many pods read only mini the volume can be mounted read-only by many pods and lastly read write once the volume can be mounted as read write by a single part and the other parts will be in read-only mode useful if you have a main worker that writes data and the other pods simply read the data so so so you first start defining a storage class specifying the cloud provider driver with the provisioner property and additional settings in the parameters section so refer to the source provider documentation on how to create a specific storage class further storage service you then Define a PVC specifying an access mode and the source capacity required in this claim uh the claim is for 5 gigabytes but more PVCs can be created over that storage class then simply refer to the PVC in your path definition and map a local folder on it in summary the main benefits of a storage class versus a purchasing volume is that with a storage class you don't have to define a capacity and multiple Claims can be made here's a cheat sheet for storage class at Men's so you create your storage class using a yaml file using Cube CTL apply Dash F and your the name of your yaml file you get a list of your storage classes or PVCs using get the SC for search class and get PVC you get the search as information by using cubectl described as C and the class name you delete your search class and PVC using cubectl delete Dash F and the yaml file name or you delete your search class using delete SC and the class name or delete PVC and the PVC name and this concludes this section about persistence using the dynamic way [Music] let's see how to store configuration values using config Maps in a previous lecture we saw that it was possible to place configuration values directly in the environment section of a pod definition but what if we need to change that value well we have to edit the Manifest and redeploy the container also usually it's not a best practice to tie an object with its configuration so how can we externalize these values the config map object allow you to decouple and externalize configuration values the key value pairs are then injected in the containers as environment variables they can be created from yaml files a series of text files or even folders containing files they are static meaning that if you change the value the containers that refer to these values have to be restarted to get these refresh values using a yaml file you define a config map and place the key value pairs in the data section you can even specify multi-line values using the pipe character in the EnV section environment section of the container definition you define an environment variable and by using value from an config map key ref ref ref you reference the config back name and the key as defined in the config map so in the config map key ref section name refers to the config map name and key refers to a key in the config map earlier I mentioned that this is a static process meaning that the values are injected when kubernetes starts the container this means that if you make a change to a config map value inside the container the original values stay the same until you restart the container to get around this you can map a volume on a config map Yes you heard it right mounting a volume this solves the static issue and updates are reflected in containers each key value pair is seen as a file in the mounted directory so we start with a config map then use a volume to mount it to a local folder inside our pod our container the result is that all key value pairs are now scenes as file the name of the file being the key and the value being inserted in the file while this sounds cool it also means that you'll have to refactor your code so instead of reading environment variables you'll have to read files so is it worth it you'll have to figure out that by yourself here's the config Maps cheat sheet so if you're adventurous you can create a config map from the command line that's the imperative Way by using Cube CTL create config map you give it a name and with the from Dash literal parameter you specify the key value pairs so you can specify multiple key value pairs of the same line or you can use a good old yaml file and use cubectl apply Dash F and the name of your yaml file you can create a config map using cubectl Create CM specify your name and specifying a the name of a text file containing multiple key value pairs also you can create a config maps from a folder so if you have multiple files inside your folders you can create a config map from that you can get a list of the config maps by using cubectl getcm you can output the config map in a yaml file by using cubectl get CM the name of the config map and the dash o parameter with the yaml and you can pipe that to to a file name of course you delete a config map by using its yaml file using cubectl delete Dash F and the name of the yaml file and this concludes this uh look at config Maps [Music] in this app we're going to create a config map and use a pod that will reference a value stored in that config map map map let's take a look at the config map so the kind is config map we have a name cm-example and in the data section we have two key value pairs date as set to Michigan and City and our board all right so let's now take a look at our pod tiny spot uh it's a busy box and here in the environment section we declare in environment variable that we will call City and we are getting the value from a config map key ref right and we specified the name of the config map so cm-example that's the name of the config map and the key is City so here that's the key right there okay again environment section we Define a new environment variable that we will call City we get the value from config map key ref specifying the name of the config map and the key awesome let's create our config map okay Cube CDL gets cm to get information about our config map so CM example two data okay that doesn't give us much information well about the data itself so let's do a cube Studio describe config map CM example okay we have the name the namespace where it is any labels annotations and here we have uh the data section City and Airport State Michigan and if for some reason you want to Output that as a yamo uh Cube CDL get config map the name of the config map Dash o output in yaml uh to recreate the config map using this let's now deploy the pod or BusyBox perfect let's open a session Cube CDL exec my box Dash it and the program you want to run and let's display the city environment variable let's Echo that Echo dollar sign City and there it is an arbor so that worked foreign exit and we can do a little bit of cleanup we can delete our config map and we can delete our busy box spot [Music] let's see how to use the secrets object in kubernetes you will find many times of Secrets types the default one is the OPAC type and it is very similar to the config Maps object that we saw in the previous lecture you can also store credentials to connect to private container registries authentication secrets and even certificates in this lecture we will focus on the OPEC Secret like config Maps secrets are used to store configuration values so they are somewhat identical to config Maps except that these store values as base64 encoded strings and it's important to understand that base64 is a way to encode strings and it is not an encryption algorithm this means that Secrets stored in kubernetes can be decoded quite easily yeah great since these secrets are Noah very secret should you use them well the answer depends on the type of information you want to store it might be okay to store a connection to a database but it might not be for something more sensitive you can protect Secrets using role-based access control policies are back or you can store them elsewhere all Cloud providers offered ways to store secrets in Seeker Vault services that you can retrieve firm kubernetes you can also use a third party tool like the very popular Vault product from archicup just be aware that the kubernetes default secret the OPEC one is not encrypted by default in kubernetes so you can define a secret in a manifest and use a base64 encoded strings as the values or use the command line where you can use plain text strings easier in the path definition you simply get the secret value using the secret key ref section this is very similar to config Maps and again similar to config maps you can mount a volume on top of Secrets here's the container registry secret and you can Define it using a yaml file or with the CLI next in the path definition you reference the credentials in the image pull Secrets section here's a cheat sheet for Secrets commands so you can create a secret the imperative way at the common line if you want using cubesdl Create secret generic and then the secret name and you pass the key value pairs with the from Dash literal per meter you can of course create one using a yaml file so cubesdl apply Dash F and the name of the yaml file you can get a list of the secrets using cubectl gets secrets you can output the secret to a yaml file by using cubectl get Secrets the secret name and the dash o yaml parameter and you can pipe that to a a file you can delete the secret using a yaml file or using the secret name using cubectl delete secrets and with the secret name and this concludes this look at secrets [Music] in this lab we'll create a secret and from a pod we will reference secret and use them as environment variable let's take a look at our secrets at yaml file the type or the kind is secret we give it a name secrets and in the data section we have key value pairs so username to some value and password to some value notice that the values but must be base64 encoded you cannot put a non-encoded string here how do you do that on Windows base64 the the tool is not installed by default so you can use these two websites basics of foreign code.org decode.org or you can install base64 using chocolati as shoko installed base64 on macro Linux well it's already installed so you simply do something like that you Echo your string and you pipe that to base 64. and that will encode the string and let me copy that and to decode Echo uh the encoder string and you pipe that again to base64 Dash D for decode and voila all right from the Pod now let's take a look at our pod yaml definition kindness part it's a busy box and here in the environment variable we're creating two environment variable the first one is called username and it gets its value from a secret key ref and the name references the secret name here and the value well the key is one of these two so username or password Here a case key reference here the username from the secrets Secret and second one is password a the environment variable is called password and we get the value from the secrets secret and the key is password awesome let's create the secret all right let's get a list of the secrets here's our secret if it has two data it was created six seconds ago we can describe it cubicity of describe the object type and then it name and here what do we have the name namespace and well we don't see the secret just the the keys here and what if we use Cube CTL get secrets and output that to demo the secret data password say uh uh uh doing a describe would not allow us to see the values but using get secret and outputting that to a yaml will allow us to retrieve the actual values okay let's now deploy our busy box right let's open a session and let's Echo uh username D username and the password whoops Echo uh uh uh password my password so that that works the uh the values the secrets are decoded when they're injected into into the pods let's exit that and let's delete our secret and let's delete our busy box [Music] let's talk about observability if you deploy a container using a deployment and it crashes kubernetes will create a brand new instance of the pod pod pod this works because kubernetes monitors the infrastructure but what about your application if your app crashes well kubernetes will look at the Pod health and see that it's still running so from the infrastructure point of view everything is working fine your bud is still up and running but your code inside the Pod has crashed would it be nice if kubernetes could monitor the application health well you can achieve this by configuring probes the startup probe informs a kubernetes that the container has started and it's now okay to send traffic to it the Readiness probe enforced kubernetes that the container is now ready to accept traffic let's say that when your app starts it needs to execute a series of steps like getting some configuration values from a database creating some files and so on and so on and let's say that this startup sequence takes around 30 seconds so even if your container is up and running your code is not ready to accept traffic so using a Readiness probe you tell kubernetes to wait 30 seconds before starting to send traffic lastly the liveness probes tell kubernetes if your app is still running and if not kubernetes will kill the Pod and replace it with a brand new one here's a part definition with the three possible probes the starter probe tell kubernetes to wait for 10 seconds before making an HTTP call to a page called health the failure threshold tells kubernetes to try three times the Readiness probe tell kubernetes to wait initially five seconds before probing and making a TCP call on port 8080 and then check every 10 seconds the liveness probe tell kubernetes to wait initially for 15 seconds before probing by making a TCP call on port 8080 and then check every 20 seconds note that the Readiness probe will run during the whole pod life cycle but will these two Conflict at some point yes and no they will run simultaneously but a fail probe will result in different actions from kubernetes failing a Readiness probe will tell kubernetes to stop sending traffic to the pod but the Pod is still alive right while failing a liveness probe will tell kubernetes to restart the pod how does kubernetes probe the containers the cubelet will do the probing using the method you configure with exec action kubernetes will run a command inside a container with TCP socket action kubernetes check if a TCP socket port is open and with HTTP get action kubernetes perform nhdp get here's an exec action you're telling kubernetes to run a cat command on a file called healthy in the TMP folder here's a TCP socket action you're telling kubernetes to check if there's an open socket on port 8080 and finally an HTTP get action you're telling kubernetes to do an HTTP get on the Health page on Port 8080.

and this concludes this look at observability [Music] in this lab we will set a liveness probe so let's take a look at our yaml file we will deploy a pod that we'll call liveness Dash example is going to be a PC box and this is where we're setting the probe type is liveness Pro we're asking to do an exacto so to run a command inside our container and the command is cat and the parameter is that file so under the TMP folder the file is called healthy no extension basically we're asking kubernetes to run that command if that command is successful if the file exists well the probe is successful if the file doesn't exist the probe will fail okay uh initial delay seconds we're asking kubernetes to wait for five seconds before starting to probe and then period seconds to five we're asking kubernetes to probe every five seconds the last parameter is failure threshold set to two basically we're telling kubernetes that the liveness probe will fail when two probes will fail now for uh the purpose of this lab we will set something a little bit funky uh just so we're able to do this test we're um we're running a command when the container starts uh it's dutch and basically this will create that healthy file and then we're telling the container to wait for 15 seconds and then to delete that file right and just a little trick too so we'll be able to to quickly see the effect of the lavenous probe okay Let's uh deploy our pod and let's quickly do a cube CTL describe pod all right successfully pull the image the container was created okay so let's do that again a few times and we'll see what happens just rerun the command out liveness profile cat can't open okay two times over five seconds and now kubernetes is killing the Pod awesome and uh is still killing so you see that here the the Pod was in an unhealthy State here so now it's in the killing mode oh and then uh kubernetes is pulling again the BusyBox image and starting a new one right and then it's on LT again and the process starts uh starts again all right let's do our cleanup let's delete our pod and we'll force it you don't want to wait for it to end perfect [Music] let's take a look at some dashboards while it's fine to use a terminal to get a view of your cluster You may wish to use a graphical user interface instead luckily there are many options available we will take a look at these three popular and free options the kubernetes dashboard the lens desktop application and K9s a dashboard that runs in the terminal let's start with the kubernetes dashboard it's a web UI that you can install as an add-on inside your cluster it's not installed by default by Docker desktop and also by most Cloud providers the rule of thumb is if you don't need it don't install it a because it runs inside the cluster so you need to find a way to expose it over the internet and B well because of that it's a known Vector of attack that being said the kubernetes dashboard that you see the various resources deployed inside your cluster simply select the type in the left menu and you can also edit them by clicking on the edit icon and you can edit the yaml file and click on update to change the Manifest file lens is a IDE that runs locally on Mac windows and Linux so you need to install it on your OS on top of viewing the resources and editing the yaml Manifest you can also use a built-in editor and also a built-in terminal lens is maintained by mirantis and you can download it from this URL here's the overview dashboard that lets you see a resource count like the kubernetes dashboard you can select a resource click on the edit button and it did the Manifest directly in lens you can also type commands using the built-in terminal K9s is a cool text dashboard that runs in a terminal and you can install it on Windows Mac and Linux it might sound strange to run the dashboard in a terminal but this makes a lot of sense Kina NS is super light starts in an instant and gives you a clean view of all your resources you can get the information about the cluster the resources and you can take a or make a series of action like deleting a resource viewing the logs here we have a list of the pods currently running in the default namespace want to view the services that are currently running simply type colon and type SVC to list the objects type foreign pressing s while a part is selected will open a shell you can also set a port forward by typing shift f you can even see the part logs I really like this K9s dashboard in a terminal and use it all the time and this concludes this look at the kubernetes dashboards thank you [Music] in this lab we'll take a look at Lens so lens is a free dashboard that you can install on Windows Mac and Linux here's the URL k8s k8s k8s lenss.dev let's take a look at the website from there you can install it it runs on Mac windows and Linux so you can download the setup files here or if you're using a security on Windows you can use shoko install lens or Brew on Mac and Brew install cast lens all right I've already launched a lens here and by default it should take you to your Docker desktop cluster if not or if you want to select another cluster click on the hamburger menu to the left and select add cluster Lance will look at the cube config file and you'll be able to select your clusters from that drop down list so here I have my demo cluster in the cloud so I can select that and click on ADD cluster I can be connected to multiple clusters here's my Docker desktop here's my demo so let me switch back to dockerdist up here okay so first thing first uh let's deploy something on our cluster here I have a yaml file it's a deployment we'll have three replicas of an image called Hello app okay nothing fancy here let's deploy this all right and let me switch back to uh lens here now let's click here on workloads and overview so we should see three parts running here one deployment and one replica set I can either from the top menu select path or from this workload menu select paths here and the other type of objects here so here are my three parts I have one deployment and one replica set if I click on an object I get more information labels annotations and so on and so on so so so something that you would have at the terminal at the common line by typing Cube CTL describe uh replica set or deployment or pod and the object name here but here is presented in a nice UI all right let's take a look at our pods here okay let's see what we can do let's say we want to delete this one so I can select it like this and from the Ellis menu to the right I can open that and select remove right or I can remove it right there okay let's do that remove item hello yep remove it and since it's a deployment uh kubernetes will create a new One automatically pretty cool uh let's uh take a look at the logs we can see the logs from there we can open a shell so at last a pretty cool front directly from that UI if I click on edit well I can edit the Manifest file and click on Save and close that will update that object let me cancel that close close this there's a built-in terminal so by default you have a small icon here called terminal if you click on it if you click on this open button that will open the terminal so I can type command Cube C Cube CTL get the pods right you can try to delete an object directly here so let me copy this object name Cube CTL delete uh and let's paste that we should see right right away we see uh something happening uh at the top of the screen all right uh what's cool is that if you have more than one cluster you can switch between these clusters so here let me uh just to prove it clear and the qctl uh get the nodes okay on Docker desktop I have only one node let me select My Demo cluster my terminology open it there's no deployment yet but let's just do a cube CTL get nodes and for sure I have three nodes in this in this cluster that's pretty cool just by selecting the cluster I can switch and I have a terminal that is in the right context pretty cool so I just scratched the surface there's a lot more information that we can get like a configuration the config Maps the secrets what the network services that are installed here I have a cluster IP service storage so the persistent volume claims persistent volume storage classes so here in my cluster in the cloud I have four search classes defined for me the namespaces and so on and so on all the information that you would get from the command line using cubectl you get the same information but from a nice UI so once you're done exploring don't forget to delete your deployment thank you [Music] in this lab we will use K9s a super great dashboard running inside a terminal so you can get more information about the K9s at the website k9scli.io here's the website nice logo and you can look at the documentation and the how to install K9s from that website if you're on Windows you can install it using chocolaty so Chico install K9s on Mac OS Brew install K9s and on Linux take a look at the documentation all right Let's uh first deploy something in our clusters here in our yaml file I have a simple deployment with three replicas of a simple container or image called Hello app nothing fancy here let's deploy that uh right away okay and let's open a terminal or come in line comment prom whatever you name it it it and let me type K9s all right let me stretch that a little bit well it's super cool it's a dashboard running inside the terminal it's super light and it gives you tons of functionality so here I have my deployment so I'm in the default namespace you can look at my my deployment I can enter on an object I get more information type Escape go back I have information about my cluster uh I can issue some some commands so Ctrl D I can delete a a pod let me do that let me kill that poor part here uh are you sure yes see you have feedback visual feedback of of uh what's happening the Pod that I deleted was kill was in the terminating State and the new one was uh created uh right away I can yeah it D to describe the resource escape to uh to go back Ctrl K to kill I just deleted one but I can do a Ctrl k there we go I can see the log so let's let's switch to the second one type l and I can look at the uh the logs I can open a shell also by typing here the the the S letter doing the less right you can type exit go back I can uh even configure a port forward pretty cool here I can look at the yaml file also let me type Escape here we have the parts that are listed but if I type Colin I can change that let's say let me type deploy so here's my deployment now if I type e I will edit the deployment and let's change the number of senses or replicas from three to four and let's close that right and now I have four out of four so let's uh I'll type Colin again and type but here are my four pods super interesting it's a free tool it's super light it's super fast and I always have one open so I can see visually what's happening uh inside my cluster when I'm issuing some some commands let's go back uh to uh our page here our Visual Studio code and let's simply delete our deployment and let me switch back right away here and they're gone pretty cool too [Music] let's see how to scale pods the horizontal pad to scalar is a kubernetes feature that allows you to scale the number of Parts up and down it uses the metric server to gather the pods utilization pods app must have requests and limits defined the HPA checks the metric server every 30 seconds and scale the number of PODS according to the minimum and maximum number of replicas defined to prevent racing conditions the HPA Waits some period of time after a scaling event by default this delay on scale-up events is 3 minutes and the delay on scaled down events is 5 minutes in this part definition you specify the CPU and memory requests and limits the request is what's allocated at first and the limit is what you allow the part to burst to in this example the Pod will start with 64 megabyte of ram but can burst up to 128 megabyte if needed you configure the HPA using a manifest specifying the deployment you want to scale the Min and max number of replicas and the metric you want the HPA to scale on in this case we tell the HPA to kick in when the average CPU utilization is above 50 percent here's the cheat sheet for the HPA commands so you can create one using the imperative way using cubectl autoscale deployment the name and the metric and replicas number you can create one using the yaml file you can get the autoscaler status by using cubectl get HPA and the HPA name and of course you can delete the HPA using the yaml file or cube CTL delete HPA with its name foreign this lecture about scaling pods [Music] in this lab we will use the horizontal particle scalar to scale a pod for the HP it worked it needs some data some metrics coming from the metric server and by default it's not installed by Docker desktop just make sure I'll open the terminal and what we'll do we'll get a list of the parts running in the cube system namespace Accord DNS at CD Cube API server proxy provisioner no nothing that looks like metric server okay to install it you need to run this yaml file coming from this git repo on kubernetes-6 special integers group and Metric server but but but you need to make a small modification to to it let's take a look at the components that yaml file need to do you need to edit it if you download it directly from the git repo and you locate the deployment section there it is deployment and what you need to do is add this parameter a cubelet dash insecure-tls if not the metric server will not run on Docker desktop all right let's deploy our metric server cubectl apply Dash F components Dot yaml awesome what we can do we can run again Cube City I'll get pod in the cube system namespace aha metrics server there it is it might take a few a couple of minutes for the metric server to start running now that the metric server is running let's take a look at our deployment uh it's a deployment and what we'll deploy is a web server called HPA Dash example uh listening on Port 80. so it's a simple web server that will return a web page nothing fancy what we'll do uh will also deploy a busy box and from that busy box will it that web server uh in a loop and that should generate some traffic awesome let's deploy our web server let's get a list of the pods running in the default namespace awesome it's running let's enable our Auto scaler so Cube CTL Auto scale a deployment called HP Dash deployment and a metric called CPU percent and we want a minimum of one instance and a maximum of four okay let's validate that we have an HPA Cube CTL get HPA that will list all the hpas running on my cluster there's only one awesome let's now deploy our busy box and let's connect or open a session on that busy box perfect and here's our endless loop that will uh it the web server that should generate some traffic okay let's take a look at our K9s K9s K9s I have my deployment here my HPA deployment and my busy box and area we have three instances of our deployment right now awesome so the HPA worked and now what I can do is start the loop by hitting Ctrl C and I'll type exit to exit my busy box and from there I can delete my HPA but be careful when you do that I'll delete my HP here if we take a look at the deployments there's still three nothing will scale that down since the HP uh has been deleted let's delete our busy box and let's delete our deployment that should delete all three instances take a look here yep all three are terminating and optionally you can delete the metric server by using cubectl delete and the components.yaml file here [Music] we are at the end of this Docker containers and kubernetes fundamental scores congratulations you are now an official kubernetes Ninja [Applause] the next steps for you would be to deploy containers in the cloud using services from a cloud provider these courses will teach you how to do that on Google cloud and Azure and also on smaller Cloud providers like Linux and digitalocean the best part is that each offer free credit usage when creating new accounts this way you can create a managed kubernetes service in the cloud without breaking the bank if you enjoyed the course you can help me by making a small donation this is the link to my buy me a coffee page I want to say a big thank you for learning Docker and kubernetes using my course and I wish you all the best

Tom here from Launch Systems. Ever wonder what Docker containers I'm actually running in my lab? Well, today I'm going to show you my setup and why I keep most everything inside of a single Debian VM with bind mounts? Spoiler, I do it because it's easier to manage, maintain, and back up.

Now, one of those containers I rely on is from today's sponsor, Netbird. It is an open- source secure overlay networking platform that makes it easy to connect devices and networks. No clunky VPN configs, just simple secure access that helps you build and manage a zero trust architecture. Netird works great in labs and in production and even supports multi-tenency for IT pros tasked with managing multiple clients.

You can self-host the entire system or let them handle the hosting for you. Check out the link below to get started. The first thing I want to address is things people have probably already left a comment down below is why not Kubernetes right now because you know someone said you should do that and I think Kubernetes is great. It's how we solve large scalable problems.

Google uses it and many of the other hyperscalers, but I like simplicity, which is why I'm going with this approach. There's nothing wrong with Kubernetes. It's just a more complicated methodology that has its merits. And for home lab users that are pursuing a career that would be using Kubernetes or they just want to go on the personal learning adventure, by all means, go do it.

I can't tell you why not to use Kubernetes with the one exception of complexity. I look for simplicity. I like this running in a single VM. And I just want to put this statement in here just to address all those people who go, "But Tom, you have all this hardware.

Why not stretch it out as a Kubernetes cluster across all of it? And wouldn't that make more sense?" And this runs a lot of other different things, but today we're focusing on the Docker VM and what I run in it and how I run it. Now, I'm not going to walk through every detail of the install, but if you'd like me to do a video on that in the future, leave a comment down below. But the install Docker engine on Debian instructions here right in the Docker docs is what I followed.

I am running because it came out recently and I just rebuilt it just to make sure there was no issues of following this guide. Debian 13 called Trixie and the setup app repository. All these functions worked perfectly fine. The one thing I will mention and this is a minor detail and if you want to know what my video notes look like, this is what they look like here.

the DNS problem. I ran into kind of a weird issue with Trixie where Docker containers that work fine in 12 weren't working in 13 because of DNS. I changed nothing. They're the same compos files.

But I want to mention in case you're banging your head on this. If you have a instance that's not properly working with DNS but has routing. Uh adding this line fixed it. You just create a Etsy/doccker/damon.json.

Edit it if it already exists. For me, it didn't exist. I had to add it and put in DNS. Now, it by default I put in Google DNS here, but probably you want to use your local DNS and that's what I changed it to, but I put this in rather than mine so someone doesn't just go, "Hey, I can put in that DNS, but Tom's DNS locally didn't work on my setup." So, start with public DNS to make sure things are working.

Switch it to local DNS if you need that local name server, which might be a better idea, but it depends on the context in which you're running things. Now, my nodes tool is log and it's not public. So I put a public forum post which you'll find linked down below to that DNS entry so you can just copy and paste it not try to read something off the screen. And more importantly this is a list of the Docker services.

I'm going to show them in a moment but I'll point out that for any one of these I have linked to the Docker install guide website whichever it may be. And for ones I've already done videos on I have linked those videos. for my fresh RSS. I have the link to Fresh RSS and of course the video Apache Guacamole and a lot of other apps in here.

So if you just are looking for the list, you can stop watching here, click on this list and just go through them. Now let's talk about how I structure Docker. Everything is loaded in / SRV/doccker. And if we do an ls, you can see all of my different Docker containers.

We're go here to Engineext proxy manager and you can see data and let's encrypt. So if we go into the docker compose file, you can see it's mounted to / srv/doccker/exproxy manager data and then maps to data. This allows me to get right into any of the files as opposed to using the docker file system. I know there's going to be a lot of opinions on this.

I do this for simplicity because if I ever wanted to move manager somewhere else, I simply grab just that folder for any one of these. And inside each of these is any of the configuration data, the compose file, everything it needs to live somewhere else. So when I moved this from Debian 12 cuz I wanted to build a new Debian 13, I simply just copied and pasted these in. And because I gave the Debian 13 the same IP, everything just worked when I brought each of these Docker containers up.

Now this is my Homeware dashboard. And there's a few things extra in here that are running on my True NAS such as Meto, which is pretty cool plugin for True NASA if you need to download things. I'll leave it at that. Uh, Jellyfin I usually run on True NAS and I left it in the list because I'm actually running it here on my Docker system.

This does create a little bit of complexity, but it was part of an upcoming video that I will be doing on Jellyfin. I wanted to try it not just on shast, but also in my Docker host. And of course, if you run it in your Docker host, you need to mount your TRNA. and I choose NFS to do that.

I have found a little bit of uh bugs with it that I haven't 100% sorted out for the video because of the way the mounts work with system D and 13. Though, if you want to go down the path of running Jellyfin on there and having it mount NFS, yes, it works. And if you solve that problem, please post in my forums if you have the startup issue where you have to start it twice. I'll leave it at that because that's why that's a future video and not something I'm necessarily saying is the easiest solution right now.

It does work perfectly fine on TRAS. But coming over here, Fresh RSS, I've done a video on it. It's my news readader. I really love Fresh RSS because I don't like the news being driven by an algorithm that suggests whatever is the best clickbait versus I want the news sources I have to be concise and in some type of chronological order from valid news sources, not random ones.

I have a list, you'll find linked down below as well to launch.vide/cybernews, video/cybernews which is my news sources and my OPML file. If you're thinking about getting started with fresh RSS, you don't have to start from scratch of adding sources in there. Maybe you like some of the same news sources I have for tech news. 13 foot ladder.

Well, we'll just say that this site is really handy for u giving what Google has as a view when it crawls the pages as opposed to all the extra scripts and stuff that may be running that maybe annoy you quite a bit. I'll just leave it at that. Check it out. 13 foot's a pretty neat project.

It's kind of an extension from the 12oot.io for those of you familiar with that. Do some reading. As I said, there's a link right to the site and explains better what it does. Open Web UI is a frontend for Olama.

I've got a video on a llama, but I don't have a specific video on Open Web UI, but essentially this is going to give you that interface. And who knows what is in here. Oh, yeah. All kinds of random things uh for metrics and stuff.

if I was looking up using self-hosted AI. This allows you to interact with and share different models, being able to choose which model gets loaded and ask the questions. I really like this. We've got a video for getting started with O Lama and maybe I'll do a separate one in open web UI, but I think there's enough people that have already done them, so I don't know that I would add a ton of value to that video, but still open web UI is great.

A couple unique ones, IT tools. This is a little Swiss army knife of handy little well generators of things like SQL pretty format or UUID generator ul generator encrypt decrypt text token generators hash text from MD5 Shaw 1 etc. Sometimes when you're doing IT work you need some of these things. Uh, you can buy this person a coffee to support the project.

But I like that it's self-hosted because I hate having to link to a site and maybe I'm solving a problem that involves the internet being down and one of these tools would be handy to have. So having it local means I always have access to it. Plus, I don't worry about things I'm putting into it because they're not going to some random website. They're going to the self-hosted one and it doesn't have any storage attached to it.

So it doesn't save any of the data that goes in here. Cybersh, if you work in cyber security, you've probably heard of Cybersh. It does encoding and decoding to a lot of different formats. It's great for when you're sorting out data that may be encoded in an unusual way.

You can just start throwing different recipes together to figure out how to decode that data or sometimes just when you're sorting out things in logs that are well weird because of the way the tooling works. This is a great tool to help decipher all of that and pull data out and put it back together. Or if you have to encode data, it it does it both ways. Next, we have Net Alert X.

This is a pretty cool tool for discovery. You plug it into your network. I have a whole setup video on this. It's a great discovery tool.

Relatively easy to get set up, fully open source and free. Can do monitoring of let you know what devices are connected, new devices discovered, and can do down a monitoring and alerting. And currently, it has found 146 devices. It also is able to pull via API and different connectors other data sources and bring them all in together.

Next, we have net data. Net data is one of those tools that over the years has just gotten better. I really like it as a tool to be able to visualize what's going on, what might be causing an event because you may know when the event happened, but you need to go back in time to when the event happened at a certain time of day at like 12:55 or 1300 and go over in a nice concise way to find all the different metrics that related like what caused this problem or what was the system doing, what caused the memory increase here and was it an application and you can filter through these. I use net data across a lot of systems.

So, I also have it loaded on this system running Docker. It can do metric correlations to try to group together the events. Uh, it has the ability to zoom in. I've got a video linked where I've done this with Net Data.

I probably should do a new one because it's a couple years old and it looks much better now than it did a few years ago. Uh, but Net Data is a great open source tool for being able to monitor in real time or even have some historical data when you're doing troubleshooting, especially related to performance. Apache guacamole. I have an entire video on this.

It is a really nice tool to let you use the web interface for things like SSH administration and maybe occasionally typing your password wrong. So, back home, I've already got it connected to this one, I can do SSH administration all through a web UI. It will also do VNC. It'll also do RDP.

So, SSH, RDP, VNC, all in a web browser. and being able to jump back and forth to other servers that you can line up in here, add and build out connections. Once again, open source and self-hostable. Also very customizable.

This actually serves as a basis sometimes for other tooling that might be more commercial if you're wondering if it can be customized quite a bit. It does have the basic needs met in terms of interface, but there's a lot of enhancement that can go on top of it. And I do talk about that in my video and of course further reading at the Apache Wacamole site. Open speed test.

It's no more than a locally hosted speed test, so it doesn't have to go out to the internet. Well, unless I guess you installed this on an external server you host, but ideally use this internally. So, you can do speed testing, whether it be hardline or probably more likely, and something I've used it more often for is Wi-Fi testing. I don't want to go out to the WAN into the ISP where the bottleneck might be.

I want to know how fast the WAN is, maybe at different distances. Having a locally hosted means I've eliminated any external ISP level factors for speed testing or troubleshooting a Wi-Fi connection. Really handy little tool. The last two tools I'm going to cover are Dazzle and What Up Docker.

Dazzle. It allows you to attach to any of the running containers and see all the logs. You can do this from the command line and attach containers and look at the output, but why not do it through a nice simple web interface? Or you can look at all of them at once by clicking the link up here and it'll consolidate all of them and tell you where these logs came from.

And if you have a lot of complicated things going on, it gives you a nice single pane of glass view for all the logs. But clicking on any individual one lets you put these here. We can even do split screen between two different sections because this, for example, is a stack that has three different components in it. And maybe we need these things side by side because we're trying to figure out what's talking to what or what's not talking to what at the same time.

Dazzle is just really a handy way to do this without having to resort to the command line, which I mean I like being at the command line, but sometimes a nice little UI and being able to click it does help quite a bit. And last but not least is what up docker. Really simple tool that has a lot of options that I haven't really enabled. I mostly just use it for what you're seeing here.

I want to know if there's an update available and then I'll manually update. I know somebody's probably saying, "But Tom, why don't you have Watchtowwer on here where it's auto updating all of your Docker images." And maybe I'll do that in the future. For now, I just go to what up Docker. It lets me know that things are needing updates.

And you can build triggers to notify you or even build out a trigger to actually update these. It has a lot of options, a lot of extensibility. I have done none of that extensibility. There are guides you can find on it.

If I do build this out, I of course do a video on it. But a lot of people might be interested in it because it's just simplicity. I like the how simple it is that it just looks, watches, lets me know what needs to be updated, and then I can just do a compose pull from the command line and uh pull the latest version as needed. And always make sure you have a backup before you do that.

Now, the last thing I want to mention is exactly where this lives as in what hypervisor am I using? I run this Docker container in XCPNG. It is my place that I like hosting things for virtualization. You may like other tools.

Proxmbox is great. You can do the same thing with Proxmbox. I just really like the integrated backups, the automated validation testing that XCPNG has. So, it doesn't just back it up because who cares about a backup that works and says successful?

We want a restore that works and this will do that full validation. So, I'm comfortable having it here. But use whatever hypervisor makes you happy. The couple exceptions, as I noted, there's a few things that actually run directly on Trunass.

And uh I still think Jellyfin and other media streaming tools are probably best suited to run on Trunass, but that's I'm still doing some experimenting. And as I said, leave some comments and maybe I'll do a Jellyfin video of running it in its own separate container versus running it on a TRASS. If you got thoughts, questions, or a different take on today's topic, drop them in the comments down below. Want to support the channel in other ways?

You can do that through Patreon or by grabbing items from our swag store. We've got some fun tech themed shirts, stickers, and other nerdy gear that won't boost your bandwidth, but will definitely boost your style. Don't forget to like, subscribe, and hit that notification bell so you don't miss future videos. If you're looking to connect with me or learn more about the services we offer, just head over to lawrence.com.

You'll find all my links to the socials and the best ways to get in touch. If you found this video helpful, chances are you'll find the next one helpful, too. So, keep learning and keep clicking wherever those videos are showing up around me. Thanks.

[Music]