[{"content":"In this project, I fine-tuned Visual Language Models (VLMs) and create an end-to-end pipeline for Handwrittent Text Recognitition tasks on historical Swedish manuscripts. The VLM-based pipeline is also compared against the classical YOLO - TrOCR pipeline.\nThe project was done in collaboration with the Swedish National Archives (Riksarkivet).\nModels: Florence-2, YOLO, TrOCR Tools: PyTorch, transformers, Gradio GitHub Demo ","permalink":"http://hoanghapham.github.io/projects/vlm-htr/","summary":"\u003cp\u003eIn this project, I fine-tuned Visual Language Models (VLMs) and create an end-to-end pipeline for  Handwrittent Text Recognitition tasks on historical Swedish manuscripts. The VLM-based pipeline is also compared against the classical YOLO - TrOCR pipeline.\u003c/p\u003e\n\u003cp\u003eThe project was done in collaboration with the Swedish National Archives (Riksarkivet).\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003e\u003cstrong\u003eModels:\u003c/strong\u003e Florence-2, YOLO, TrOCR\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eTools:\u003c/strong\u003e PyTorch, transformers, Gradio\u003c/li\u003e\n\u003c/ul\u003e\n\u003cdiv class=\"github-link-wrapper\"\u003e\n  \u003ca href=\"https://github.com/hoanghapham/vlm-htr\" class=\"github-button\" target=\"_blank\" rel=\"noopener noreferrer\"\u003e\n    \u003ci class=\"fab fa-github\"\u003e\u003c/i\u003e GitHub\n  \u003c/a\u003e\n\u003c/div\u003e\n\u003cdiv class=\"generic-link-wrapper\"\u003e\n  \u003ca href=\"https://huggingface.co/spaces/nazounoryuu/vlm-htr\" class=\"generic-link\" target=\"_blank\" rel=\"noopener noreferrer\"\u003e\n    \u003ci class=\"fa-solid fa-laptop-code\"\u003e\u003c/i\u003e Demo\n  \u003c/a\u003e\n\u003c/div\u003e","title":"Master's Thesis - HTR with Visual Language Model"},{"content":"This project simulates a monitoring system for kitchens, using YOLO for object detection and tracking. The detection model is trained to track two types of items (dish, tray), and each item type has a classification model to further categorize them into three sub-classes, depending on the content of the dish or tray.\nThe system includes a module to perform inference, and a module for users to re-annotate the object detection themselves.\nModels: YOLO (for detection, tracking, and classification) Tools: FastAPI, Gradio, Docker GitHub ","permalink":"http://hoanghapham.github.io/projects/kitchen-monitoring/","summary":"\u003cp\u003eThis project simulates a monitoring system for kitchens, using YOLO for object detection and tracking. The detection model is trained to track two types of items (dish, tray), and each item type has a classification model to further categorize them into three sub-classes, depending on the content of the dish or tray.\u003c/p\u003e\n\u003cp\u003eThe system includes a module to perform inference, and a module for users to re-annotate the object detection themselves.\u003c/p\u003e","title":"Kitchen Monitoring"},{"content":"In this project, I trained two models for the language detection task, using the WiLI-2018 dataset.\nModels: Naive Bayes, XLM RoBERTa Tools: sklearn, transformers, Streamlit, FastAPI, Docker. GitHub ","permalink":"http://hoanghapham.github.io/projects/language-detector/","summary":"\u003cp\u003eIn this project, I trained two models for the language detection task, using the WiLI-2018 dataset.\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003e\u003cstrong\u003eModels:\u003c/strong\u003e Naive Bayes, XLM RoBERTa\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eTools:\u003c/strong\u003e sklearn, transformers, Streamlit, FastAPI, Docker.\u003c/li\u003e\n\u003c/ul\u003e\n\u003cdiv class=\"github-link-wrapper\"\u003e\n  \u003ca href=\"https://github.com/hoanghapham/language-detector\" class=\"github-button\" target=\"_blank\" rel=\"noopener noreferrer\"\u003e\n    \u003ci class=\"fab fa-github\"\u003e\u003c/i\u003e GitHub\n  \u003c/a\u003e\n\u003c/div\u003e","title":"Language Detector"},{"content":"With this project, I aim to combine the real Million Song Dataset with data coming from Spotify API. With this pipeline, new researchers that are already familiar with the old Million Song Dataset to get new information for all the songs in Spotify, like new audio features, song popularity, artist popularity\u0026hellip;\nTools: Amazon S3, Redshift, EC2, Prefect, Terraform\nGitHub ","permalink":"http://hoanghapham.github.io/projects/music-etl/","summary":"\u003cp\u003eWith this project, I aim to combine the real Million Song Dataset with data coming from Spotify API. With this pipeline, new researchers that are already familiar with the old Million Song Dataset to get new information for all the songs in Spotify, like new audio features, song popularity, artist popularity\u0026hellip;\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eTools:\u003c/strong\u003e Amazon S3, Redshift, EC2, Prefect, Terraform\u003c/p\u003e\n\u003cdiv class=\"github-link-wrapper\"\u003e\n  \u003ca href=\"https://github.com/hoanghapham/music-etl\" class=\"github-button\" target=\"_blank\" rel=\"noopener noreferrer\"\u003e\n    \u003ci class=\"fab fa-github\"\u003e\u003c/i\u003e GitHub\n  \u003c/a\u003e\n\u003c/div\u003e","title":"Music ETL"},{"content":"You can reach me via LinkedIn, or drop me at email at hapham.elv@gmail.com.\n","permalink":"http://hoanghapham.github.io/contact/","summary":"\u003cp\u003eYou can reach me via \u003ca href=\"https://www.linkedin.com/in/hoanghapham/\"\u003eLinkedIn\u003c/a\u003e, or drop me at email at \u003ca href=\"mailto:hapham.elv@gmail.com\"\u003ehapham.elv@gmail.com\u003c/a\u003e.\u003c/p\u003e","title":"Contact"},{"content":"I\u0026rsquo;m an aspiring Machine Learning Engineer / AI Engineer with 5+ years of industry experience in data analytics.\nRecently completed my Master\u0026rsquo;s degree in Language Technology at Uppsala University, I\u0026rsquo;m looking for opportunities to improve my skills in machine learning, NLP, and AI engineering.\nAside from coding, I enjoy reading, writing, gaming, movie-going, and playing music.\nTed Chiang\u0026rsquo;s \u0026ldquo;Stories of Your Life\u0026rdquo; is my all-time favorite sci-fi story.\n","permalink":"http://hoanghapham.github.io/about/","summary":"\u003cp\u003eI\u0026rsquo;m an aspiring \u003cstrong\u003eMachine Learning Engineer / AI Engineer\u003c/strong\u003e with 5+ years of industry experience in data analytics.\u003c/p\u003e\n\u003cp\u003eRecently completed my Master\u0026rsquo;s degree in Language Technology at Uppsala University, I\u0026rsquo;m looking for opportunities to improve my skills in machine learning, NLP, and AI engineering.\u003c/p\u003e\n\u003cp\u003eAside from coding, I enjoy reading, writing, gaming, movie-going, and playing music.\u003c/p\u003e\n\u003cp\u003eTed Chiang\u0026rsquo;s \u003ca href=\"https://www.goodreads.com/book/show/223380.Stories_of_Your_Life_and_Others\"\u003e\u0026ldquo;Stories of Your Life\u0026rdquo;\u003c/a\u003e is my all-time favorite sci-fi story.\u003c/p\u003e","title":"Hi, I'm Ha."},{"content":"Introduction As someone transitioning from an analyst to a data engineering role, I struggled to find a standardized road map to follow, since there were so many resources to learn from, and it was very easy to get lost. After trying out a few free courses and also crafted my own learning path, I decided to take on the paid Udacity Data Engineering Nanodegree last September just to know if it has anything better than the free resources. Although I don\u0026rsquo;t think this program is a waste of time, I\u0026rsquo;m not entirely convinced that it worths every penny.\nIn this review, we will take a closer look at what this nanodegree offers, what benefits it provides, and whether it is the best option for you to obtain data engineering knowledge.\nProgram structure Basically, this nanodegree program is a compilation of of existing courses in Udacity, with the aim to cover necessary topics to start a Data Engineering career. Of course, all of the course contents are pre-recorded videos delivered online.\nThe courses have several lessons to introduce you the basic concepts, as well as the tools to implement those concepts. For example, in the Data Modeling course, you will learn to build database in PostgreSQL and Apache Cassandra, and in the Cloud Data Warehouse course you will get to learn about AWS and Redshift.\nAt the end of each lesson, you will demonstrate your understanding of the newly acquired knowledge with a small project. After that, Udacity\u0026rsquo;s reviewers will check your project\u0026rsquo;s code, evaluate it against the project\u0026rsquo;s standard (rubric), and give comments on how you can improve your code. As of the time of my enrollment, most of the exercises revolved around building data pipelines for a fictional music streaming service using the Million Song Dataset, and you will handle a very small subset of the real dataset.\nFinally, to graduate from the course you will need to complete a capstone project in which you will build a data pipeline from end to end. You can either choose from a project they provided, or find a topic or a dataset of your own interests\nFor the final project, I decided to process the real Million Song Dataset because at the time, I was too busy to think of a new topic.\nHow expensive is it? As of the time of this writing, the program costs either $399 per month, or $1356 for a four-month access. However, you can claim a coupon by clicking on the \u0026ldquo;Your Personalized Offer\u0026rdquo; that significantly reduces the amount you have to pay. When I enrolled to the course in September 2022, I paid about $500 after applying a 70% coupon, and I had a five-month access to the course materials. Looks like the price has gone up a little bit.\nPersonally I think the price is ridiculously high for some pre-recorded videos of people reading slides.\nSo does the program really worth that much?\nMy feelings about the course If you are a beginner, you may like it In case you have just pivoted into a data engineering role, this program will give you an outline of the topics that you need to know to perform in the role. Having a structured program to follow gave me an ease of mind, because the DE field is vast, and having a clear road map to study is quite comforting.\nAnother unique thing about this program is that for the end-of-course projects, you will have Udacity\u0026rsquo;s reviewers to review and comment on your code, unlike other courses that rely on peer reviews. However, I was not so sure about the quality of the review. To me it looks like they just review your code basing off a list of check boxes. In some cases, I was sure that my code is quite shitty but they still praise me to the moon.\nOther neat features of the course include the opportunity to try out AWS services for free, and an electronic certificate to attach to your Linkedin profile after the course, if you care about that kind of thing.\nIf you already know your stuff, then just skip this While I enjoyed some courses in the program, I can\u0026rsquo;t say that this entire nanodegree is well-constructed.\nI really liked the pipeline automation course because it was closed to what I had been working on in my day job, and I wanted an Airflow refresher. On the other hand, the remining courses are pretty boring. I find the content pretty shallow, since most of the time they only focus on introducing the tools, while I expected to learn more about the principals of designing data systems.\nThe content delivery is also inconsistent, because the program is comprised by different courses taught by different instructors. Some instructors\u0026rsquo; delivery is serviceable, while some of them are just plain boring, like they just read everything off the slides. This made me wonder why I have to pay money for this at all, since they are no different from the kind of videos that I can easily find on Youtube.\nMoreover, the logistic of the classroom is also not very good. The project workspaces that Udacity provided all use out-dated software packages from 2018, so if you want to do the projects on your local machine, you have to make sure to use the same python package versions as the ones installed on Udacty\u0026rsquo;s workspace, so that the code reviewers can run your code.\nFinal verdict So, should you take this course? It really depends. I\u0026rsquo;d say you should take this if these are true for you:\nYou are new to data engineering, and you are so overwhelmed by the field that you want to have some hand-holdings You do not have the patient to go through loads of blog posts and Youtube videos to compile your own learning program. You have money to spare, or your company pays for the course On the other hand, if these apply to you, probably you should not take this program:\nYou already have some experience doing data engineering. You know where to look for information, and you are confident that you can craft your own learning roadmap You do not want to learn out-dated technologies You do not want to pay for this course yourself, and there is no one sponsoring you. Recommendations If you want a good free alternative data engineering course, I think DataTalksClub\u0026rsquo;s DE Zoomcamp is a good choice. If you register at the right time, you can join the instructors and other learners via Zoom on a weekly basis. This gives the feeling that you are really attending a class room with real people and real interactions, which is a very big plus in my opinion. If you cannot attend the live sessions, the recordings are also available. Another exciting aspect of this Zoomcamp is that the instructors update the course content regularly and often incorporate new tools into the stack, so you can rest assured that what you learn are pretty up-to-date with what engineers out there are using.\nSo that\u0026rsquo;s my opinion about the Udacity Data Engineering Nanodegree. Hope that this review will help you decide if you should take this program or not.\n","permalink":"http://hoanghapham.github.io/posts/udacity-de-nanodegree-review/","summary":"\u003ch1 id=\"introduction\"\u003eIntroduction\u003c/h1\u003e\n\u003cp\u003eAs someone transitioning from an analyst to a data engineering role, I struggled to find a standardized road map to follow, since there were so many resources to learn from, and it was very easy to get lost. After trying out a few free courses and also crafted my own learning path, I decided to take on the paid \u003cem\u003eUdacity Data Engineering Nanodegree\u003c/em\u003e last September just to know if it has anything better than the free resources. Although I don\u0026rsquo;t think this program is a waste of time, I\u0026rsquo;m not entirely convinced that it worths every penny.\u003c/p\u003e","title":"Udacity Data Engineering Nanodegree Review (2022)"},{"content":"What is run-length ID? Sometimes during analysis work, you need to group consecutive sequence of values into different \u0026ldquo;runs\u0026rdquo;, and calculate metrics for each run. For example:\nGiven a time series recording some entity\u0026rsquo;s state, you want to calculate on average, how long does an entity stay in a particular state. Or a more specific example: Given a series of event data of several users, you want to group the users\u0026rsquo; events into sessions with a session cut-off threshold of your choice This technique is also called sessionization.\nGenerate run-length ID with R If you are an R user, you may have heard of this kind of analysis. This operation is made trivial with the data.table package, since its rleid/rleidv function can directly generate run-length IDs that are incremented when a new run is commenced.\nSince I work most of the time with SQL, sometimes I need to have this kind of behavior in my SQL code. After some experiments, I finally figured out a way to do it. Here\u0026rsquo;s how:\nFirst, indexing all the rows in the dataset Next, figure out the \u0026ldquo;points\u0026rdquo; where the state of your entity changes from one to another by comparing current row and a previous row. ID these \u0026ldquo;switch points\u0026rdquo;. Use the switch points\u0026rsquo; IDs as the ID of the consecutive runs In this post I will use BigQuery\u0026rsquo;s Standard SQL to demonstrate how this work, but you can easily implement this in other SQL flavors, as long as your SQL support analytical functions (LEAD, LAG, ROW_NUMBER, FIRST_VALUE, LAST_VALUE\u0026hellip;)\nGenerate run-length ID with SQL Example: How long does an object stay in a state? Suppose we have a device with two state (active/inactive) and we periodically record the state of the device at a particular time. We want to calculate how long do they typically stay in a certain state\n1. Index all rows, and get the previous state The first step is to create a column containing the previous state, and a \u0026ldquo;row index\u0026rdquo; column. We will use the previous_state field to determine the point where the device switch to a new state, and use the idx field to generate the run-length ID.\nSince we have multiple devices, we have to partition by device_id when using lag() to make sure we do not get the state of a device into data of another device.\nselect * , lag(state, 1) over (partition by device_id order by timestamp) as previous_state , row_number() over (order by timestamp) as idx from test.consecutive_runs order by device_id, timestamp 2. Determine the state switching points Next, we will check for the \u0026ldquo;switch point\u0026rdquo;. If an event is a switch point, we will make the idx of that event NULL. This is to prepare for the generation of run-length ID.\nwith base as ( select * , lag(state, 1) over (partition by device_id order by timestamp) as previous_state , row_number() over (order by timestamp) as idx from test.consecutive_runs order by device_id, timestamp ) select * , case when previous_state is null or state != previous_state then idx else null end as switch_points from base As you can see, row 1 (start of the event series), row 3, row 7 and row 11 are marked as the \u0026ldquo;switch points\u0026rdquo;.\n3. Filling in the NULLs Next we can use the IDs at these switch points to fill the NULL rows below, up to the next switch point. This forward-filling feature can be achieved by BigQuery\u0026rsquo;s last_value() function:\nwith base as ( select * , lag(state, 1) over (partition by device_id order by timestamp) as previous_state , row_number() over (order by timestamp) as idx from test.consecutive_runs order by device_id, timestamp ) , switching as ( select * , case when previous_state is null or state != previous_state then idx else null end as switch_points from base ) select * , last_value(switch_points ignore nulls) over (order by timestamp) as run_length_id from switching So that\u0026rsquo;s how we have our run-length IDs. Now we can check on average how long each run lasts:\n... , aggr as ( select device_id , run_length_id , timestamp_diff(max(timestamp), min(timestamp), second) / 60 as session_duration from run_length_table group by 1, 2 ) select device_id , avg(session_duration) as average_session_duration from aggr group by 1 4. Another way to generate run-length IDs In step 2 and 3 above, we made use of NULLs and BigQuery\u0026rsquo;s last_value() function to generate the IDs. There is another way to generate these IDs using 1, 0 and a moving sum().\nInstead of marking \u0026ldquo;switch points\u0026rdquo; using idx and set the following row values to NULL, we can mark the switch point using value 1, and set the following rows to 0:\nwith base as ( select * , lag(state, 1) over (partition by device_id order by timestamp) as previous_state from test.consecutive_runs order by device_id, timestamp ) select * , case when previous_state is null or state != previous_state then 1 else 0 end as switch_points from base Then, use sum(...) over (partition by ...) to perform a running sum of switch points. Whenever the run hits a 1 value, the ID will be increased by one and will stay the same for the next rows (because the next rows are all 0).\nwith base as ( select * , lag(state, 1) over (partition by device_id order by timestamp) as previous_state from test.consecutive_runs order by device_id, timestamp ) , switching as ( select * , case when previous_state is null or state != previous_state then 1 else 0 end as switch_points from base ) select * , sum(switch_points) over (partition by device_id order by timestamp) as run_length_id from switching Final result: Practical use cases for sessionization This technique can have multiple applications, for example:\nYou have records user traffics on your website, but you are not happy with your tracker\u0026rsquo;s default session grouping. You may want to generate the sessions with your own logic. Your have a complex web application, and you have a specific workflow that you want users to go through. You may want to group user\u0026rsquo;s actions into workflows, and check the ratio of finished over unfinished flows. Hope that this simple tutorial can give you ideas to solve these kinds of problem.\n","permalink":"http://hoanghapham.github.io/posts/generate-run-length-id-with-sql/","summary":"\u003ch1 id=\"what-is-run-length-id\"\u003eWhat is run-length ID?\u003c/h1\u003e\n\u003cp\u003eSometimes during analysis work, you need to group \u003cstrong\u003econsecutive sequence of values\u003c/strong\u003e into different \u0026ldquo;runs\u0026rdquo;, and calculate metrics for each run. For example:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003e\n\u003cp\u003eGiven a time series recording some entity\u0026rsquo;s state, you want to calculate on average, how long does an entity stay in a particular state.\n\u003cimg loading=\"lazy\" src=\"/generate-run-length-id-with-sql/consecutive_runs.png\"\u003e\u003c/p\u003e\n\u003c/li\u003e\n\u003cli\u003e\n\u003cp\u003eOr a more specific example: Given a series of event data of several users, you want to group the users\u0026rsquo; events into sessions with a session cut-off threshold of your choice\n\u003cimg loading=\"lazy\" src=\"/generate-run-length-id-with-sql/user_events.png\"\u003e\u003c/p\u003e\n\u003c/li\u003e\n\u003c/ul\u003e","title":"Generate Run-Length ID With SQL"},{"content":"If you are familiar with the modern data stack, probably dbt is no stranger. dbt tries to bring the best practices from the software engineering world into data development, and one of such practices is the idea of automated testing and continuous integration (CI).\nWhile dbt Cloud provides a \u0026ldquo;slim CI\u0026rdquo; feature that satisfies most basic needs, you will have more control over your CI jobs if you make use of your git provider\u0026rsquo;s CI/CD functions. In this project, we will look at how to create a dbt CI job using GitHub Actions\nThe demo project can be found here: https://github.com/hoanghapham/dbt_ci_demo\nWhat is GitHub Action? GitHub Actions is GitHub\u0026rsquo;s tool to schedule \u0026amp; execute software development workflows right within your GitHub repository. The workflows are configured using YAML files placing in the .github/workflows folder.\nIf you are not familiar with the tool, it is best to start with GitHub\u0026rsquo;s documentation first. This tutorial will assume that you have already grasped the basic concepts of GitHub Actions.\nPreparation Required dbt version: 0.21.0 or above (As of this writing\u0026rsquo;s date, v0.21.0 is the latest version). dbt 0.21 has introduced the powerful dbt build command that you should definitely use.\ndbt project setup: dbt stores information about database connections in the profiles.yml file. In this tutorial, we will place the file in a test_profiles folder.\nDatabase: In this tutorial, I will use a free Google BigQuery account. You can easily register for one yourself following this instruction. If you decide to use BigQuery, you will also need to create a Service Account, and download the key following this instruction.\nActions secrets: Important information like credentials, passwords, access tokens\u0026hellip; must not be committed to the repository. Instead, you can set them up as Action secrets.\nWith that out of the way, let\u0026rsquo;s dive in.\nRun a workflow upon PR creation First, let\u0026rsquo;s configure a job to run upon a new PR against the main branch.In general, the workflow will need to have the following steps:\nCheck out the code branch you have just pushed Read the database credentials from the repository\u0026rsquo;s secret Install dbt to the GitHub action runner, as well as necessary packages Build \u0026amp; test dbt models Archive the compiled SQLs for debugging purpose Suppose that our profiles.yml has a ci target like so:\ndemo_ci: target: ci outputs: ci: type: bigquery method: service-account project: \u0026#34;{{ env_var(\u0026#39;DBT_CI_BIGQUERY_PROJECT_ID\u0026#39;) }}\u0026#34; dataset: dbt_ci keyfile: ./test_profiles/cred.json threads: 4 timeout_seconds: 300 priority: interactive Our basic workflow will look like this:\nname: CI testing - Full run - No container on: [pull_request] # Run this workflow when there is a new PR jobs: ci-full-run-no-container: runs-on: ubuntu-latest env: DBT_PROFILES_DIR: ./test_profiles # Tell dbt to look for profiles in this folder DBT_CI_BIGQUERY_PROJECT_ID: ${{ secrets.DBT_CI_BIGQUERY_PROJECT_ID }} # Make the BigQuery project ID available as an env var steps: - name: Check out ${{ github.head_ref }} uses: actions/checkout@v2 - name: Read Bigquery credentials from repo secret shell: bash env: DBT_CI_BIGQUERY_CRED: ${{ secrets.DBT_CI_BIGQUERY_CRED }} run: | mkdir -p ./test_profiles; echo $DBT_CI_BIGQUERY_CRED | base64 -d -i \u0026gt; ./test_profiles/cred.json # Need to install dbt into the runner - name: Install dbt \u0026amp; packages shell: bash run: | pip3 install dbt==0.21.0; dbt deps; - name: Build \u0026amp; test models shell: bash run: dbt build # Upload compiled SQL as artifacts - name: Archive compiled SQL if: ${{ always() }} uses: actions/upload-artifact@v2 with: name: compiled_sql path: ./target/compiled A few notes:\nAfter the job finishes running, the runner will be destroyed. All resources created in the runner (checked-out codes, new files created\u0026hellip;) will also be destroyed, unless you upload them as artifacts.\nIn the Read credential step, since I encoded my credentials using base64 before adding it into the secrets, I needed to decode it before putting it into a cred.json file.\nIn the final step (Archive compiled SQL), the if: ${{ always() }} expression ensures this step will always run, even when the previous dbt build step fails (because of a failed test, or an invalid dbt run). This step will make the compiled SQL available to you after the job finished running.\nAfter merging this workflow, whenever you make a new PR against your main branch, the workflow will be triggered. The GitHub UI will show you which workflows are running, just like when you enable Slim CI in dbt.\nIncremental testing workflow The problem with the workflow above is that, even when you only modify one model, the whole project will be rebuilt when you invoke dbt build. This can result in an expensive (as in, cost you more BigQuery money) and slow-running workflow.\nTo avoid this, you can use dbt\u0026rsquo;s state and defer feature to compare your current project with a previous state, and only run the new and modified models.\nThe state of a dbt project is reflected in the manifest.json file. When you run some dbt commands (like compile, run, test\u0026hellip;) this file will be generated into the target folder. By default, this folder is ignored and not pushed to GitHub, so we need to make this file available in the action runner.\nThere are a few approaches, each of which has its own pros and cons:\nManually commit the manifest.json file Generate the manifest.json file at run time Have an automated workflow to generate and commit the manifest.json file After having the manifest file available, you can change the dbt build step into\ndbt build --select state:modified --defer --state folder-with-manifest-file/ Let\u0026rsquo;s go into the details of the approaches below.\nManually commit the manifest file You can either:\nAdd !target/manifest.json into the .gitignore file so that everything in the target folder will be ignored except the manifest.json file. This way, every time you run a dbt command and the state of your project changes, you can commit that change.\nThis is also the drawback of this approach, since during model development, it is unnecessary to commit all the tiny changes. In the GitHub Actions workflow, you will also need a step to copy the manifest file to a different location other than the target folder before running a new dbt command.\nOr, manually copy the manifest.json file to a different location only when necessary. This way you have better control of which state to retain, but of course, you have to remember to do it every time you push something new to your repository.\nThe advantage of this approach is that it is the easiest to do. However, it is quite cumbersome, and is definitely not cool. We are here to do cool stuff, so let\u0026rsquo;s automate this process.\nAutomatically generate manifest file at run time One way to have the manifest file reflecting the old project state is to generate it directly from your main branch. Here\u0026rsquo;s the workflow to do so:\nname: CI testing - Incremental run - Checkout master on: [pull_request] jobs: ci-incr-run-checkout-master: runs-on: ubuntu-latest env: DBT_PROFILES_DIR: ./test_profiles DBT_CI_BIGQUERY_PROJECT_ID: ${{ secrets.DBT_CI_BIGQUERY_PROJECT_ID }} steps: - name: Check out ${{ github.head_ref }} uses: actions/checkout@v2 # Check out master branch to a different folder - name: Checkout master uses: actions/checkout@v2 with: ref: master path: master_branch/ # Should also copy the credential into the master_branch folder - name: Read Bigquery credentials from secret shell: bash env: DBT_CI_BIGQUERY_CRED: ${{ secrets.DBT_CI_BIGQUERY_CRED }} run: | mkdir -p ./test_profiles; echo $DBT_CI_BIGQUERY_CRED | base64 -d -i \u0026gt; ./test_profiles/cred.json; echo $DBT_CI_BIGQUERY_CRED | base64 -d -i \u0026gt; ./master_branch/test_profiles/cred.json; - name: Install dbt \u0026amp; packages shell: bash run: | pip3 install dbt==0.21.0; dbt deps; - name: Generate manifest.json from master shell: bash run: dbt compile --project-dir master_branch/ --profiles-dir master_branch/test_profiles/ # Tell dbt to look up previous manifest file in master_branch/target - name: Build \u0026amp; test models shell: bash run: dbt build --select state:modified --defer --state master_branch/target/ - name: Archive compiled SQL if: ${{ always() }} uses: actions/upload-artifact@v2 with: name: compiled_sql path: ./target/compiled How this workflow is different from the previous one:\nFirst, we check out the files from our master branch to the master_branch folder within the current project Then, we generate the manifest file from the master branch by running dbt compile, while specifying the project directory as master_branch/. The manifest file will be generated into master_branch/target/ folder. Note that it is necessary to also copy the credential file into the master_branch folder. Finally, run dbt build while pointing to the master_branch/target/ folder for state comparison As you can see, in the incremental workflow, only the new test_model.sql file is run, while the original full run workflow will run all the files.\nSo we have started to automate the boring stuff! However, this workflow has a drawback. Every time you push something new to an opened PR, the whole checkout and generate manifest steps will have to run again.\nThis may be OK if your project is small, but it may cost you more job run time if your project has hundreds of models.\nWe can certainly flex further and look for a way to reuse this manifest file.\nGenerate manifest file when merging to main branch During model development, it is unlikely that the manifest file will change that much. We would want to reuse this manifest file, but GitHub Actions does not allow sharing files between different job runs.\nWe can work around this with workflow that update the manifest file when you merge a new PR. This is the workflow that do so:\nname: Update dbt project state on: pull_request: types: [closed] workflow_dispatch: jobs: update-project-state: if: github.event.pull_request.merged == true runs-on: ubuntu-latest env: DBT_PROFILES_DIR: ./test_profiles DBT_CI_BIGQUERY_PROJECT_ID: ${{ secrets.DBT_CI_BIGQUERY_PROJECT_ID }} steps: - name: Checkout master uses: actions/checkout@v2 with: ref: master - name: Read Bigquery credentials from secret shell: bash env: DBT_CI_BIGQUERY_CRED: ${{ secrets.DBT_CI_BIGQUERY_CRED }} run: | mkdir -p ./test_profiles; echo $DBT_CI_BIGQUERY_CRED | base64 -d -i \u0026gt; ./test_profiles/cred.json; - name: Install dbt \u0026amp; packages shell: bash run: | pip3 install dbt==0.21.0; dbt deps; - name: Generate manifest.json from master shell: bash run: | dbt deps; dbt compile --no-version-check; mkdir -p ./current_state; cp ./target/manifest.json ./current_state/manifest.json; - name: Commit new manifest.json file uses: EndBug/add-and-commit@v7.4.0 with: add: \u0026#39;./current_state/manifest.json\u0026#39; message: \u0026#39;manifest.json updated\u0026#39; push: true branch: master This part:\non: pull_request: types: [closed] workflow_dispatch: jobs: update-project-state: if: github.event.pull_request.merged == true runs-on: ubuntu-latest specifies the types of events that will trigger this workflow.\npull_request: This workflow will run when you merge a PR to the master branch (defined with the if expression).\nworkflow_dispatch: This means that you can also manually trigger this workflow from GitHub Action\u0026rsquo;s UI.\nAfter this, in the main CI workflow you can remove the \u0026ldquo;Checkout master\u0026rdquo; and \u0026ldquo;Generate manifest file from master\u0026rdquo; steps. Since we now have the manifest.json file in the current_state folder, when running dbt build you need to point to this folder for state comparison.\ndbt build --select state:modified --defer --state current_state/ Bonus: Run jobs inside a container In the workflows above, we have a step to install dbt into the runner:\npip3 install dbt==0.21.0 You can also run dbt using dbt Labs\u0026rsquo; official Docker image so you won\u0026rsquo;t need to worry about dependencies. Simply add the container and image properties to the job config:\njobs: ci-incr-run: runs-on: ubuntu-latest container: image: fishtownanalytics/dbt:0.21.0 Now you can remove the pip3 install dbt==0.21.0\nConclusion Hope that this demonstration can help others who are trying to improve their dbt workflow. If you have any comments or questions, please create an issue in this repo: https://github.com/hoanghapham/dbt_ci_demo.\n","permalink":"http://hoanghapham.github.io/posts/dbt-ci-with-github-actions/","summary":"\u003cp\u003eIf you are familiar with the modern data stack, probably \u003cstrong\u003edbt\u003c/strong\u003e is no stranger. dbt tries to bring the best practices from the software engineering world into data development, and one of such practices is the idea of automated testing and continuous integration (CI).\u003c/p\u003e\n\u003cp\u003eWhile dbt Cloud provides a \u0026ldquo;slim CI\u0026rdquo; feature that satisfies most basic needs, you will have more control over your CI jobs if you make use of your git provider\u0026rsquo;s CI/CD functions. In this project, we will look at how to create a dbt CI job using GitHub Actions\u003c/p\u003e","title":"How to run dbt CI with GitHub Action"},{"content":"You can read Part 1 here.\nFull code to produce the charts and report: https://github.com/hoanghapham/vietnam_war_bombing\nThere are many notable operations happened during the Vietnam War: Rolling Thunder, Steel Tiger, Barrel Roll, Line Backer, Line Backer II… However, I chose to explore Rolling Thunder because of its interesting nature: the bombing strategy of the operation changed over time due to the U.S. policy against China and the Soviet.\nThe evolution of the operation After Geneva conference in 1954, the U.S. replaced France as a political backup for Ngo Dinh Diem in the South Vietnam. They believed in the “domino effect”, which suggested that if Vietnam fell under the influence of communism, all countries around would follow.\nAt first, the U.S. believed South Vietnam’s government could be grown into a self-sustained one. However, by the beginning of 1965, that belief turned into the realization that “without further American action, Saigon government could not survive.”\nOperation Rolling Thunder was devised as a demonstration of the U.S.\u0026rsquo;s commitment to Saigon. Officially it lasted from March 2nd,1965 to November 2nd, 1968 with three main objectives:\nTo boost Saigon’s morale while not directly and severely affect Hanoi. To persuade North Vietnam to stop supporting the South’s communists without sending ground force into the North To destroy North Vietnam’s transport system, industrial base, air defenses, which, in effect, will cut off the supporting line from North to South. The U.S. wanted to chase North Vietnam off the South, but they did not dare to go for a full-blown air campaign lest China or Soviet would retaliate with their full might. Therefore, their bombing strategy slowly escalated and changed over time, which will be visualized below.\nRoute packages To minimize airspace conflict among air forces, North Vietnam was divided into six target regions called “route packages” (RP).\nEach region was assigned to one air force or navy, and they were forbidden to intrude each others’ regions. The Navy’s Carrier Task Force 77 handled operations in RP 2, 3, 4, and 6B, as these bordered on the Gulf of Tonkin. The Air Force was given RP 1, RP 5, and 6A. We can see this division clearly from the bombing targets map:\n1965 — Testing the water At the beginning of the operation, Washington believed in “gradualism”, which meant “the threat of destruction is a better signal of U.S.’ determination than the destruction itself”. This translated into the the U.S. strategy to “hold important target hostage by bombing trivial ones”. Johnson tightly controlled the campaign and refused to attack Hai Phong port directly as he deemed it “too provocative”.\nWe can see this initial hesitation clearly when looking at bombing map of 1965:\nOne notable bombing strategy of this time was the focus on POL (Petroleum, Oil, Lubricant) targets. Since the beginning of the operation, this was one of the top target type that the military pushed for Johnson’s approval, as they believed depriving the North of POL would effectively halt their war efforts.\nRegrettably we do not have data of 1965, but we can still see POL targets were bombed with much higher tonnage in the beginning but then went out of focus in the later years. The reason was that by the time POL target were approved by Johnson in June 1965, North Vietnam’s army had spread POL reservoir across the whole country. POL bombing were halted after two months, when the U.S. realized that their attacks had no effect.\n1966–1967 — Escalation According to air force historian Earl Tilford:\n“(Initially) Targeting bore little resemblance to reality in that the sequence of attacks was uncoordinated and the targets were approved randomly — even illogically. The North’s airfields, which, according to any rational targeting policy, should have been hit first in the campaign, were also off-limits.” 1\nThis course of action angered the military generals. They dissatisfied with the randomness and uselessness of the targets, and complained that striking and re-striking some targets benefited the North Vietnam’s defense force. Thanks to the repetitiveness of targets, Vietnamese gunners got time to adapt to U.S. patterns and incurred heavy lost on its air forces.\nTo calm these concerns, from mid 1966 to 1967 President Johnson approved attacks on sensitive targets, especially targets in Hanoi and Hai Phong.\nDuring 1966 there was a huge spike in the number of “armed reconnaissance” missions. This tied to military’s complaints about the faulty target request system. As summed up by this post on Quora, it took too much time for the Air Force and Navy to request and get approvals to attack a target. By the time the target was approved, it had already moved or became trivial.\nLooking at the target types of these missions we can further understand their functionalities. While targets of Strike missions were fixed, Armed recon missions’ targets were more likely to be moving (like vehicles). This type of mission made use of small aircraft formations to patrol highways, railroads, rivers and bombed whatever they deemed a good target.\n1968 — The bloodiest year of the war In 1968, Rolling Thunder reached its final stage of operational evolution. Its purpose transformed from psychological warfare to interrupting the logistics flow from the North to the South.\nIn the following map, we can see most of the targets were in the the “pan-handle” area, with Ha Tinh suffered the most from the U.S. escalated bombing attacks.\nThe North’s military weapon sites (anti aircraft, missile launchers…) as well as vehicles were bombed more heavily this year. This is both because of the escalation of war, as well as the need to stop weapons and supplies flowing from North to South.\nBombing tonnage increased year-by-year and peaked in 1968:\nThe increase in intensity also reflected in tonnage over area. To calculate the bombing area, I got longitude and latitude degrees at 10th and 90th percentiles of their distributions, calculate the differences in degrees and convert it to kilometers. For simplification, I approximated the range of one latitude / longitude to 111km.\nWith this calculation, we can see during the peak bombing time of 1968, each square kilometer received more than three times the amount of bombs and weapons in the previous peaks. This was both due to the increase in bombing tonnage, and the higher focus on specific areas.\nPutting everything together:\nAircrafts usage analysis A large portion of the missions during Rolling Thunder were classified as Armed Recon and Strike. Which aircrafts were used in these types of mission?\nThere were many aircrafts of different application used in the war. However, we will focus on Attack and Fighter-bomber types.\nAttack aircrafts were favored, Bombers’ time was over Attack aircrafts were used to carry out air strikes with greater precision than bombers. As we already know, Rolling Thunder’s targets evolved from random locations to more “sensible” ones, so we should expect Attack aircrafts’ use increased over time while Bombers’ usage declined.\nDouglas A-1 Skyraider\nThe A-1 Skyraider was an attack aircraft transferred to RVNAF and remained as its “close air support workhorse” for much of the Vietnam War. Its usage in the U.S. forces gradually decreased over time and was replaced by the A-6A Intruder and A-4 Skyhawk. This was a part of the general switch to jet aircraft at that time.\nFighter-bombers: F4 \u0026amp; F105 A fighter-bomber is a fighter aircraft that has been modified or used primarily as a light bomber or attack aircraft, whereas bombers and attack aircraft were developed specifically for their respective roles. Due to this versatility, fighter-bomber type was a mainstay in the U.S. Army.\nAt the same time, dogfighting (air-to-air combat, which is one of the main use of fighters) had fallen out of favor in the U.S. This was due to their belief that missiles were all they needed to shoot down Soviet’s heavy bombers, and official dogfight training only returned by 1969 with the TOPGUN program.\nIn the graph below, we can see pure fighter’s usage is lower than that of fighter-bomber — probably due to some fighters were converted to fighter-bomber.\nMcDonnell Douglas F-4 Phantom II\nU.S. Navy were mainly responsible for the development of the F-4 and then distributed it to the USAF and USMC, expecting it to have great performance.\nWe can clearly see the F-4’s growing usage over time. On the other hand, the F-105 was later removed from combat due to high loss rates.\nConclusion Operation Rolling Thunder was intended to be a \u0026ldquo;warning\u0026rdquo; from the U.S. to the North Vietnam, but in the end, it was not effective. There are thousands of reasons for the operation\u0026rsquo;s failure, but one very interesting detail that I came across when researching for the post is the U.S.’ underestimation of Vietnamese army’s intelligence capabilities.\nWith the slow escalation of war, Vietnamese army had time to adapt to the U.S. new strategies. Vietnamese signals intelligence staff of 5,000 were also “proved adept at exploiting traffic analysis as NSA was”, when they deduced that before every bombing mission had an upsurge of traffic involving logistics and recon flights. Interestingly, they were also able to intercept U.S. radio communication, which used unencrypted voice. This is a major source of early warning for the Vietnamese army.\n“…captured documents showed that the North Vietnamese had at least thirty to forty-five minutes’ warning of 80 to 90 per cent of Rolling Thunder missions.” 2\nThis severe security problem were not unknown to the U.S. Air Force, but they ignored NSA’s request to install voice encryption on aircrafts. (At this point I wished that there are data related to this, but perhaps one can’t ask for too much.)\nThis concludes my current investigation into the Vietnam War using data. Hope that in the future I will come across other interesting datasets like this.\nOperation Rolling Thunder: The History of the American Bombardment of North Vietnam at the Start of the Vietnam War\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nWikipedia: Operation Rolling Thunder\u0026#160;\u0026#x21a9;\u0026#xfe0e;\n","permalink":"http://hoanghapham.github.io/posts/vietnam-bombing-history-p2/","summary":"\u003cp\u003eYou can read Part 1 \u003ca href=\"http://hoanghapham.github.io/posts/vietnam-bombing-history-p1/\"\u003ehere\u003c/a\u003e.\u003c/p\u003e\n\u003cp\u003eFull code to produce the charts and report: \u003ca href=\"https://github.com/hoanghapham/vietnam_war_bombing\"\u003ehttps://github.com/hoanghapham/vietnam_war_bombing\u003c/a\u003e\u003c/p\u003e\n\u003chr\u003e\n\u003cp\u003eThere are many notable operations happened during the Vietnam War: Rolling Thunder, Steel Tiger, Barrel Roll, Line Backer, Line Backer II… However, I chose to explore Rolling Thunder because of its interesting nature: the bombing strategy of the operation changed over time due to the U.S. policy against China and the Soviet.\u003c/p\u003e\n\u003ch1 id=\"the-evolution-of-the-operation\"\u003eThe evolution of the operation\u003c/h1\u003e\n\u003cp\u003eAfter Geneva conference in 1954, the U.S. replaced France as a political backup for Ngo Dinh Diem in the South Vietnam. They believed in the “domino effect”, which suggested that if Vietnam fell under the influence of communism, all countries around would follow.\u003c/p\u003e","title":"Vietnam bombing history with data - Part 2: Rolling Thunder Operation"},{"content":"History as taught in Vietnam schools is boring. Modern war history is even more boring, because of the very unattractive way textbooks present the narrative of war. We were taught that our army is brave, noble and great, and we had impossible feats considering the size and technology level of our country. However, I am always skeptical about all those teachings. History as told by only one side is never complete, and I want to know what “the other side” can tell me about the war.\nIn this two-part series, I use the THOR (Theater History of Operations Reports) dataset to explore the bombing history of my country. This dataset, which was painstakingly compiled by Lt. Col. Jenns Robertson in more than 8 years, contains the last 70–100 years of bombing data. The database has already proved useful in finding unexploded bombs in South East Asia.\nIn this first part, I will provide an overall narrative of the war because this is also something that I like to know. No in-depth analysis is involved.\nData Description For the full dataset, you can follow the link above to download. You can find my data cleaning \u0026amp; reporting code here: https://github.com/hoanghapham/vietnam_war_bombing\nThe original data set consists of 4,8 million rows describing each run. As defined in the data dictionary, one aircraft delivering a particular weapon or strike a particular target will generate a new record. The data contains information about sorties like: operation supported, mission type, aircraft used, weapon (bomb) used, military services carrying out the mission, target coordinate, tonnage of weapons delivered…\nThe data is compiled from paper reports, so it is expected to have problems — in other words, the data is not “clean”. Some problems:\nDuplicated sorties: the data is compiled from many sources, and there are cases where the data is updated. It is unavoidable to have duplicated records.\nNon standardized operation / mission naming: great efforts have been spent to standardize the mission names and fix typos. However, this is just me doing it and I have little history and military knowledge, so I’m not sure if I fixed them correctly.\nIncomplete data: As of the time of this writing, no data prior to September 30th 1965 were included. The data structure is still a work in progress, and this introduces another problem to the analysis, as there are columns that I cannot understand their meanings.\nDefinition of some terms:\nStrike: correspond to one row in the dataset. If the same aircraft carrying the same weapons but attacked two targets, this will be counted as two strikes Operation: The OPERATION_SUPPORTED field in the dataset is quite messy. It seems that the operation names are comprised of the name of the whole operation and the number assigned to identify the missions. Why the war happened at all? The official time span of the war is 1955–1975, but the root of it started a bit further. France’s rule in Vietnam lasted for about 60 years from 1887 and ended with Vietnam’s proclamation of independence in 1945. However, France did not accept Vietnam’s independence, so between 1945 - 1954 they tried to reestablish themselves to no avail. It was in this period that the U.S. got involved in Indochina affairs.\nDuring Japan\u0026rsquo;s occupation of French Indochina in WWII, Viet Minh was established in 1941 as an organized resistance group seeking to free Vietnam. Along with fighting France, Viet Minh also opposed Japan, so it received supports from the U.S., Soviet Union and China. After Japan’s surrender in 1945, Viet Minh fought against France to protect Vietnam’s new-found freedom.\nIn 1949, France established the State of Vietnam under the nominal rule of King Bao Dai in the south, rearing it to be an opposition force against communist North Vietnam. At the same period, Domino Theory was popular in the West and stated that “if one country in a region came under the influence of communism, then the surrounding countries would follow in a domino effect”1. The U.S. saw Viet Minh’s affiliation with communist ideas as a great danger, thus gradually turning their support to France. Even after France’s defeat in 1954, they still fixated on the idea that South Vietnam should not be a communist state.\nBelieving that Ngo Dinh Diem has the potential to drive Viet Minh away from the South, the U.S. supporting him overthrow Bao Dai’s government and provided him with military services. From this moment it turned out that the U.S. drew the short straw. South Vietnam’s government was reported to be “corrupted and unpopular”, and is a difficult state to support. No matter how many “advisors” were sent to train South Vietnam’s forces, they could not defeat Viet Cong.\nFinally, after the infamous Tonkin Gulf incident in 1964, the U.S.’s moved to an actively offensive stance in South East Asia.\nVienam war period 1965–1973 I used the article Vietnam War: Escalation and Withdrawal through rare photographs, 1968-1975 as the basis to conduct my investigation, meaning I will go through main ideas and demonstrate it using data.\nSome general statistics of Vietnam war’s bombing from end of 1965 to 1975:\nNumber of operations: 639 Number of aircraft type used: 190 Number of weapons type used: 293 Countries targeted: North Vietnam, South Vietnam, Thailand, Laos, Cambodia, Phillipines and some West Pacific locations Total tonage of bombing was 7,255,140 tons while the whole tonnage of U.S. bombing in World War II was roughly 2,057,244 — three times less than that in Vietnam. We can have an overview of the U.S.’s escalation and withdrawal in Vietnam by looking at the number of air strikes throughout the years.\n1965–1968: bloody years, and the beginning of peace talk U.S.’s involvement increased gradually from 1965 and topped in 1968. This is a bloody year for both sides of the war, and is considered the transition from the “idealism” of the 1960s and the “disillusionment” of the 1970s. Due to Tet Offensive, U.S. press and public started to “challenge the Johnson administration’s assurances of success and to question the value of the increasingly costly war.”2\nThe graph below shows lessened air force activities after 1968, with another come bank in 1972 when peace negotiations broke down. We will come back to this year later, as it is a very interesting period to look into.\nData of U.S. force’s fatality in Vietnam War further cement this fact. In 1968 about 16,000 U.S. soldiers were reportedly deceased (in the article, this number is 14,000). The number decreased gradually in the after years.\nIn March 1968, Johnson decided that U.S.’ effort in Vietnam could no longer be justified. After being requested 200,000 more men, he consulted with his new secretary of defense and outside advisors and decided that a limit had been reached. Johnson authorized only 13,500 more and informed Thieu and Ky that South Vietnam will “have to carry more of the fighting”. This later results in what was called “Vietnamization” of the war.\nIn general, the U.S. Air Force is the main actor. Other main military services took part included U.S. Navy, Marine Corps and (South) Vietnam Air Force.\nThere were other air forces from Laos, Australia, Khmer and U.S. Army, but their roles are not significant.\nOver the years U.S. forces still hold the main responsibility to carry out air strikes. However, after 1968 U.S. forces’ activities lessened and VNAF carried more runs.\nNext we will look at the bombing target maps across the years, as it also reflects the changes of the U.S.\u0026rsquo;s war policy.\nWe can clearly see the North was bombed quite heavily throughout 1965–1967, as well as on the famous Ho Chi Minh Trail.\nOn 31st March 1968 Johnson announced on TV that the United States would “restrict bombing of North Vietnam” and pursue negotiation with Hanoi. We can see that bombing in the north lessened in 1968 and stopped in 1969–1970.\n1969–1972: stagnant in negotiation and miscommunications In 1969, Nixon became president with a promise to end the war. He ordered more and more U.S. soldiers to withdraw, and tried to push Vietnamization strategy but saw little progress.\nHowever, facing political pressure at home and the army’s dissatisfaction in the front, starting from 1970 Nixon sent U.S. troops to the neutral Cambodia, as this had been considered the “sanctuary” of Viet Cong and North Vietnam Army that the U.S. dared not to touch.\nIn 1971, to further support Vietnamization, heavy air attacks rained down on communist supply lines in Laos and Cambodia. We can clearly see bombing targets shifted to Cambodia and Laos, as well as area around 17th parallel.\nIn 1972 U.S. bombed Hanoi and Hai Phong again. This is the year of “Christmas bombing”, which is a series of bombings which was considered “heaviest in the war to date.” The reasons for this escalation is multi-layered, and as far as I understand, this is a huge confusing mess. After some research, here is my synopsis of the situation then.\nNorth Vietnam and the U.S. had been in secret peace negotiations since 1968 but had several deadlocks. For three years, North Vietnam maintained their requirement that the U.S. needed to bring Thieu down and replace him with someone more “acceptable” in the North’s point of view. At the same time, the U.S. demanded North Vietnam Army to withdraw completely from the South. In 1972, Le Duc Tho and Henry Kissinger in Paris finally made progress in the negotiation. The U.S. accepted a cease-fire as a precondition for its withdrawal without requiring the North to do the same.\nHowever, the situation quickly became chaotic as this agreement was made without the knowledge of Nguyen Van Thieu. When Thieu was presented with the draft of that agreement, he was furious and refused to accept it. On October 24th 1972, Thieu made a broadcast “emphasized that the South Vietnamese could not agree to the Communist proposal for ceasefire in place before a political settlement.” Because of this, Hanoi believed they were deceived by Kissinger. On October 26th 1972 they also broadcasted the key details of the agreement made with the U.S.\nAt the moment Nixon was facing major pressure to bring the war to an end. He pressed Thieu to accept the agreement even though his demands would not been met. Nixon assured to provide South Vietnam supports in case the North attacks, and to demonstrate his seriousness, Nixon ordered operation Linebacker II to bomb Hanoi and Hai Phong from December 18th to 30th (thus the name “Christmas Bombing”). The purpose of this move is also to force Hanoi to stay at the table — meaning to prevent Hanoi from abandoning negotiation and seek total victory.\nDuring this operation, the U.S. also suffered the heaviest B-52 loss in the war.\nAt around Christmas (Dec 25th — 26th) Hanoi proposed a resumption of peace talk on January 8th, and U.S. bombings completely stopped on Dec 30th. However, the motive of the bombing halt is reported differently Hanoi — it claimed that this was a victory over the U.S., and the U.S. withdrew due to the loss inflicted by North Vietnam’s Army. In either way, I still think that this ceasefire can be considered a victory for Vietnamese people. In this Linebacker II operation alone, they already lost too much.\nIn this map we clearly see bombing activities intensified in Hanoi and Haiphong again.\nLet’s do a quick recap by putting all maps into a GIF image:\nIn 1973, the U.S. stopped bombing the North completely as peace talk with Vietnam in Paris achieved breakthroughs. The rest of the story is basically what you have known: on April 30th 1975 the liberation army occupied the Independence Palace, and:\n“the last remaining Americans abandoned the U.S. embassy in Saigon in a dramatic rooftop evacuation by helicopters.”1\nConclusion War story has never been a pretty one. Digging through articles to make this presentation is like opening a can of worms that can eat your souls. However, I did learn a lot about this war by digging through this data set instead of doing pure reading.\nIn the next post in the series, we will take a deeper look at Operation Rolling Thunder, one of the most important operations in this war.\nReferences https://wikieducator.org/images/8/8b/VIETNAM_WAR_BACKGROUND.pdf https://www.airuniversity.af.mil/News/Article/704552/historic-airpower-database-now-online/ https://www.u-s-history.com/pages/h1888.html https://en.wikipedia.org/wiki/Operation_Rolling_Thunder https://en.wikipedia.org/wiki/Ho_Chi_Minh_trail https://www.nytimes.com/1972/10/25/archives/speech-in-saigon-ceasefire-obstacles-seen-but-president-expects.html Domino Theory\u0026#160;\u0026#x21a9;\u0026#xfe0e;\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nVietnam War: Escalation and Withdrawal through rare photographs, 1968-1975\u0026#160;\u0026#x21a9;\u0026#xfe0e;\n","permalink":"http://hoanghapham.github.io/posts/vietnam-bombing-history-p1/","summary":"\u003cp\u003eHistory as taught in Vietnam schools is boring. Modern war history is even more boring, because of the very unattractive way textbooks present the narrative of war. We were taught that our army is brave, noble and great, and we had impossible feats considering the size and technology level of our country. However, I am always skeptical about all those teachings. History as told by only one side is never complete, and I want to know what “the other side” can tell me about the war.\u003c/p\u003e","title":"Vietnam bombing history with data - Part 1"}]