Open Source for You - September 2017

Demystifying Serverless Computing

120 ` 120

A Glimpse Of Microservices With Kubernetes And Docker

ISSN-2456-4885

Volume: 05 | Issue: 12 | Pages: 112 | September 2017

Splinter Simplifies Web App Testing

How To Secure And Test

WEB APPLICA APPLICATIONS

A Few Tips For Scaling Up Web Performance Getting Started With PHP, The Popular Programming Language

Interview: Karanbir Singh, Project Leader, CentOS

Case Study: Open Source Enables PushEngage To Serve 20 Million Push Notifications Each Day!

https://2ndQuadrant.com • [email protected] India +91 20 4014 7882 • USA +1 650 378 1218 • UK +44 870 766 7756

Experience Innov Innovation ation without lock-in!

2ndQPostgres More speed More reliability More scalability

ISSN-2456-4885

Admin 29

A Primer on Software Dened Networking (SDN) and the OpenFlow Standard

35

Taming the Cloud: Provisioning with Terraform

40

Visualising the Response Time of a Web Server Using Wireshark

42

DevOps Series Creating a Virtual Machine for Erlang/ OTP Using Ansible

47

An Introduction to govcsim (a vCenter Server Simulator)

57

A Glimpse of Microservices with Kubernetes and Docker

Developers 59

Selenium: A Cost-Effective Test Automation Tool for Web Applications

65

Splinter: An Easy Way to Test Web Applications

73

Crawling the Web with Scrapy

Serverless Architectures: Demystifying Serverless Computing

51

Using the Spring Boot Admin UI for Spring Boot Applications

61

77 Five Friendly Open Source Tools for Testing Web Applications 81 Developing Research Based Web Applications Using Red Hat OpenShift 85

A Few Tips for Scaling Up Web Performance

90

Regular Expressions in Programming Languages: The Story of C++

FOR U & ME 88

Open Source Enables PushEngage to Serve 20 Million Push Notications Each Day!

96

Eight Top-of-the-Line Open Source Game Development Tools

4

REGULAR FEATURES 06 FOSSBytes

| SEPTEMBER 2017 | OPEN SOURCE FOR YOU | www.OpenSourceForU.com

18

New Products

108

Tips & Tricks

FOSSBYTES Google launches ‘Made in India’ programme to showcase local developers Google has launched its own ‘Made in India’ initiative. The new development is designed to promote Indian developers, giving them a chance to feature their work on the Play Store in a special section. “At Google Play, we are committed to helping Indian developers of all levels seize this opportunity and build successful, locally relevant businesses,” said Purnima Kochikar, director of business development for games and applications, Google Play. Google highlights that more than 70 per cent of Internet users in India enter the Web primarily using smartphones. This growth in smartphone usage has prompted the company to encourage domestic developers to build more apps and games. More content could pave the way for the Android maker to strengthen its presence in India and around the globe. Revealing some numbers, Google underlines that Indian users on Android install more than a billion apps every month from Google Play, and this number is growing by 150 per cent each year. The ‘Made in India’ initiative was launched as a part of the App Excellence Summit that was recently hosted in Bengaluru. Google also showcased success stories from developers including Dailyhunt, Healthifyme, RailYatri and UrbanClap that are all natively building apps and services for the Android platform. Skill-building consultation sessions and demonstration booths were available at the venue for developers. Indian developers who want to participate in the ‘Made in India’ programme by Google need to ll in a self-nomination form. The apps need to be based on Google’s ‘Build for Billions’ guidelines that were launched last year.

PiCluster v2.0 brings better container management for Docker deployments Linux Toys has announced PiCluster 2.0. The new version of the open source container management tool is written in Node.js and is designed to deliver an upgraded experience through cleaner CSS and JQuery dialogue windows. PiCluster 2.0 brings automatic container failover to different hosts. It xes reported errors in npm build dependency as well as utilises enhancements on the CSS front to deliver a fresh look to the tool’s Web console. Additionally, users can deploy container management without Internet access by using the Web server to deliver required libraries. On booting up PiCluster 2.0, you’ll be welcomed with a new screen. The open source community has also contributed a lot of features to the latest PiCluster

6


Compiled by: Jagmeet Singh

Angular 5 is out, with a focus on progressive Web apps Google has released the next major version of its JavaScript framework, AngularJS. This latest version, Angular 5, is the second major update in 2017. While the initial release is a beta build of Angular 5, the search giant is clearly aiming to introduce major support for Google-driven progressive Web apps with the latest development. The new version includes a build optimiser that helps to reduce the code of progressive apps designed through the framework. Google is working hard at simplifying the effort that goes into building progressive Web apps. The purpose of this new innovation is to improve the experience for users accessing services through their mobile devices.

In addition to its progressive Web app focus, Google is integrating Material Design components into Angular 5. The design components in Angular 5 are now compatible with server-side rendering. Google is not the sole enabler to have enhanced browser-based apps. Mozilla is also set to offer a nativelike experience on its Firefox browser by bringing progressive Web apps to the front. The team behind PWAs (progressive Web apps) is working towards making these as the technology that everyone can use. The release cycle of Angular has been quite aggressive. Google plans to release the next major version, slated as Angular 6, sometime in March or April next year. Meanwhile, the theme for Angular 5 is ‘Easier, smaller, faster’.

FOSSBYTES

version. One of the initial contributors worked on the x for npm dependency errors and pm2 support. Another notable contribution improved the Web console by adding personalisation options. Previous versions of PiCluster were used to display a server icon on specic operations. However, this new build shows the operating system’s or distribution’s logo for each server. There is also an automatic container failover that helps you to automatically migrate a container to another host after three failed attempts. Many developers have started contributing to the PiCluster project. You can access the PiCluster 2.0 code through its GitHub repository. It also includes a detailed readme to help you deploy the tool effectively.

ActiveRuby debuts with over 40 gems and frameworks ActiveState, the open source languages company, has graduated its Ruby release to the rst beta version. The commercially supported Ruby distribution is supposedly far better than other available options. Ruby is actively used by a diverse set of developers around the world. The language is preferred for its complete, simple, extensible and portable nature. ActiveRuby is based on Ruby v2.3.4 and includes over 40 popular gems and frameworks, including Rails and Sinatra. There is also seamless installation and management of Ruby on Windows to reduce conguration time as well as increase developer and IT productivity. Enterprise developers can adopt the latest Ruby distribution release internally to host Web applications. The Canadian company claims that ActiveRuby is far more secure and scalable for enterprise needs. The beta release of the language has xed some issues of gem management to enhance security. The new ActiveRuby version also includes non-GPL licensed gems. All major libraries for database connectors, such as MongoDB, Cassandra, Redis, PostgreSQL and MySQL, are also included. Additionally, ActiveRuby beta introduces cloud deployment capabilities with Amazon Web Services (AWS) along with all the necessary integration features for AWS. “For enterprises looking to accelerate innovation without compromising on security, ActiveRuby gives developers the much-needed commercialgrade distribution,” said Jeff Rouse, director of product management, ActiveState, in a statement. ActiveRuby is currently available only for Windows. The release for Mac and Linux is supposed to roll out later in 2017. You can download the beta through the ofcial ActiveState website.

GNOME’s disk utility to get large file support in v3.26 Though GNOME’s disk utility will receive an update to version 3.26 in September, it is now expected to receive features such as disk resize and repair functions. The new version will also get large le support to handle giant les. The new disk utility will be launched as part of the GNOME 3.26 release. Kai Lüke, the developer of GNOME Disk Utility, has published a blog post that highlights the new features in the upcoming release. The latest version is touted to offer a le system resize. Generally, it is not possible to estimate the exact space occupied by a specic le system. So the new disk utility package will resize le systems that are in partitions. The future releases will also receive improved support for both NTFS and FAT le system resizing. The updated GNOME disk utility will also have the ability to update the window for power state changes. Additionally, the new version will prompt users when it stops any running jobs while closing an app. It will debut with better support for probing and unmounting of volumes. GNOME developers will enable an app menu entry in the new disk utility. This will help you create an empty disk image. Likewise, you will get the option to check the displayed UUIDs for selected volumes. GNOME 3.26 is scheduled to go live on September 13. You can download Disk 3.25.4, which has been released for testing. Its source tarball is available for download, and you can use it with your GNU/Linux distribution.

www.OpenSourceForU.com | OPEN SOURCE FOR YOU | SEPTEMBER 2017 |

7

FOSSBYTES

Arduino founder plans ‘sustainable’ growth Massimo Banzi, the developer of the Arduino board, has agreed to acquire 100 per cent ownership of Arduino AG, the company that owns all Arduino trademarks. The latest development is supposed to help the company generate sustainable growth through its open source hardware and software developments. “This is the beginning of a new era for Arduino in which we will strengthen and renew our commitment to open source hardware and software, while in parallel setting the company on a sound nancial course of sustainable growth,” said Banzi, in an ofcial statement. As a result of the acquisition, Banzi, 49, has become the new chairman and CTO of Arduino. The CEO, Federico Musto, has also been replaced by Dr Fabio Violante. “In the past two years, we have worked very hard to get to this point. We envision a future in which Arduino will apply its winning recipe to democratise the Internet of Things for individuals, educators, professionals and businesses,” said Dr Violante. Developed as an open source project back in 2003, Arduino is aimed at providing affordable solutions to individuals to build new devices. The boards under the Arduino range are available as open hardware and are compatible with a range of sensors and actuators. Last month, the Banzi-led company even partnered with the LoRa Alliance to start building hardware with the LoRaWAN standard. In May this year, the Arduino Foundation began to build an open source ecosystem for sectors like education, IoT markets, makers and receivers. “Our vision remains to continue enabling anybody to innovate with electronics for a long time to come,” said Banzi.

8

PayPal launches Technology Innovation Labs in India PayPal has launched two of its Technology Innovation Labs in India to support developments specic to new age technology. Located at PayPal’s Chennai and Bengaluru centres, the labs are the rst in India, opened after the Palo Alto rm launched its US and Singapore labs. “India is a hotbed for technology innovation given its evolving startup ecosystem, diverse merchant proles and enormous talent pool,” said Mike Todasco, director of innovation, PayPal. “To cater to their needs in the most effective manner, we are delighted to announce the launch of our newest Technology Innovation Labs in India, where the focus will be on fuelling new age technology and giving rise to unconventional ideas with the potential to transform the ecosystem we operate in,” he added. PayPal’s Technology Innovation Labs will support diverse elds including machine learning, articial intelligence, data science, Internet of Things, penetration testing, software-dened radios and wireless communication, virtual and augmented reality, computer vision and basic robotics. The company will provide equipment like Raspberry Pi boards with sensor kits, AlphaBot kits, Amazon Echo, LeapMotion and 3D printers, among others. “Enabling innovation and creating amazing experiences for our customers is at the heart of PayPal’s global success, and the Innovation Lab is another step to foster this spirit in our development centres in India,” said Guru Bhat, general manager — technology and head of engineering, PayPal. In addition to providing relevant hardware to kickstart the innovative developments, PayPal is set to integrate its native incubation centre. Called the PayPal Incubator, the centre was launched back in 2013 with an aim to support India-origin startups.

Google starts discriminating against poor quality Android apps Google is all set to improve the user experience on Android by enhancing its search and discovery algorithms on Play Store. This will have a direct impact on apps that have quality issues. A new Android vitals dashboard in the Google Play Console was revealed at I/O 2017 earlier this year. The technology is designed to understand and analyse inferior app behaviour such as excessive battery consumption, slow render times and crashes – and hence set benchmarks for what passes as a quality app. “Developers focusing on performance can use the Play Console to help nd and x a number of quality issues,” wrote Andrew Ahn, product manager of Google Play, in a blog post. Google reports that the change in its algorithms has shown that more users have downloaded quality apps. The Android maker also recommends developers to examine ratings and reviews that they received on their apps to get additional insights about their quality. If you are about to launch your app and test its functionality in the alpha or beta stage, you can use the pre-launch report to x issues ahead of mass downloads. Likewise, Android vitals can be applied to identify performance issues reported by opt-in devices.


FOSSBYTES

Developers ask Adobe to open source Flash Player Many developers have not welcomed Adobe’s decision to end support for the Flash Player plugin in 2020. Thus, a petition seeking the open source availability of Flash Player has been released on GitHub. While Adobe may have plenty of reasons to kill Flash, there are a bunch of developers who want to save it. GitHub user Juha Lindstedt, the developer who has led the petition, believes that Flash is an important part of the Internet’s history. Killing support for Flash means that future generations would not be able to access old games, websites and experiments, Lindstedt has said.

World’s first software-defined data centre gets launched in India Pi Datacenters, India’s native enterprise-class data centre and cloud services provider, has launched Asia’s largest Tier IV-certied data centre in Amaravati, Vijayawada. The company claims the new offering, called Pi Amaravati, is the world’s rst software-dened data centre. “Pi Amaravati is a major milestone for the entire team,” said Kalyan Muppaneni, founder and CEO, Pi Datacenters. The new data centre uses the OpenStack virtualisation framework to deliver an advanced computing, storage and networking experience. It is capable of offering league modular colocation and hosting services with a capacity of up to 5,000 racks. Also, the company’s enterprise cloud platform Habour1 is powered by open source provider SUSE. Vijayawada-based Pi Datacenters has recently been awarded Uptime Institute Tier IV design certication — known as the highest standard for infrastructure, functionality and capacity. “With the launch of Pi Amaravati, we will be offering highly innovative and tailored solutions with Infrastructure-as-a-Service (IaaS), Platform-as-a-Service (PaaS), Disaster-Recovery-as-a-Service (DRaaS) and a host of other cloud-enabled product and services to our esteemed partners,” Muppaneni said. Along with launching the Pi Amaravati data centre, Pi Datacenters has entered into a Memorandum of Understanding (MoU) with companies like PowerGrid, IRCTC, Mahindra and Mahindra Finance, Deutsche Bank and Unibic. These partnerships will expand open source developments in the data centre space.

LibreOffice 5.4 has ‘incremental’ compatibility with Microsoft Office files “Open sourcing Flash specs would be a good solution to keep Flash projects alive safely for archival reasons,” the developer wrote in the petition. Interestingly, over 3,400 people have so far signed the petition, which compares Adobe Flash with saving and restoring of old manuscripts. The developers who’ve signed the petition also want the interactive artwork created with Flash to be saved. The petition clearly states that it is not requesting Adobe to release the licensed components. Instead, the petitioners are ready to volunteer for either bypassing the licensed components or to replace them with open source alternatives.

10

The Document Foundation has released an update to the LibreOfce 5 series -- the version 5.4, which has new features for Writer, Calc and Impress. In the list of major tweaks over the previous version, the Document Foundation states that there are a large number of ‘incremental’ improvements to Microsoft Ofce le compatibility. “Inspired by Leonardo da Vinci’s belief that ‘simplicity is the ultimate sophistication’, LibreOfce developers have focused on le simplicity as the ultimate document interoperability sophistication,” said the non-prot organisation in a blog post. The Writer element of the LibreOfce 5.4 brings improved compatibility for Microsoft Word les. The ODF and OOXML les written by the LibreOfce suite are also more robust and easier to share than before. The simplicity concept translates the XML description of a new document with 50 per cent smaller ODF/ODT les and 90 per cent smaller OOXML/DOCX les as compared to Microsoft Ofce. The other highlight of the latest LibreOfce update is the new standard colour palette based on the RYB colour model. The Document Foundation has integrated better support for embedded videos and OpenPGP keys. Also, the rendering of imported PDF documents is much better in this version. The new version of Writer can help you import AutoText from MS Word DOTM templates. Users can preserve the le structure of exported or pasted lists


FOSSBYTES

Linux gets a preview of Microsoft’s Azure Container Instances Microsoft is adding a new service to its cloud portfolio dubbed Azure Container Instances. While the development is yet to receive Windows support, a public preview for Linux containers is out to help developers create and deploy containers without the hassle of managing virtual machines. Microsoft claims that Azure Container Instances (ACI) takes only a few seconds to start. The conguration window is highly customisable. Also, users simply need to select the exact memory and count of CPUs that they need. Designed to work with Docker and Kubernetes, the new service allows developers to utilise container instances and virtual machines simultaneously in the same cluster. Microsoft is also releasing ACI connector for Kubernetes to help the deployment of clusters to ACIs. “While Azure Container Instances are not orchestrators and are not intended to replace them, they will fuel orchestrators and other services as a container building block,” said Corey Sanders, director of compute, Azure, in a statement. The company executives are hoping that ACIs will be used for fast bursting and scaling. Virtual machines can be deployed alongside the cloud to deliver predictable scaling so that workloads can migrate back and forth between two infrastructure models. Windows support for ACI is likely to be released in the coming weeks. In the meantime, you can test it on your Linux container system.

12

as plain text. This allows them to create custom watermarks for their documents. Additionally, a new context menu is available to help users with footnotes, endnotes, styles and sections. The new version of Calc has support for pivot charts. Users can customise pivot tables and comment via menu commands. Impress helps users in specifying fractional angles while duplicating objects. There is also an auto save feature for settings to help in duplicating an operation. This is a part of Calc as well as Impress. LibreOfce 5.4 is available for download for Mac OS, Linux and Windows through its ofcial website. The organisation has also improved the LibreOfce online package with better performance and a more responsive layout. You can access the latest LibreOfce source code as Docker images.

OpenSUSE Leap 42.3 is out with new KDE Plasma and GNOME versions OpenSUSE has released the new version of its Leap distribution. Debuted as OpenSUSE Leap 42.3, the new release is based on SUSE Linux Enterprise (SLE) 12 Service Pack 3. The new update includes hundreds of updated packages. There is the new SUSE version that is powered by Linux kernel 4.4. The development team has spent a good eight months in producing this rocksolid Leap build. The most notable addition in OpenSUSE Leap 42.3 is the KDE Plasma 5.8 LTS desktop environment. Users have the option to either pick the latest KDE version or go with GNOME 3.20. There is also a provision to install other supported environments. Apart from the new desktop environment options, the OpenSUSE Leap update comes with a server installation prole and includes a full-featured text mode installer. The platform also ofcially supports Open-Channel solid-state drives through the LightNVM full-stack initiative. Likewise, there are numerous architectural improvements for 64-bit ARM systems. The OpenSUSE team has provided PHP5 and PHP7 support in the latest Leap distro. There is also an updated graphics stack based on Mesa 17, and GCC 4.8.5 as a default compiler. Considering the list of new changes, OpenSUSE 42.3 appears to be an advanced Linux version. It also comes preloaded with packages for streaming media, editing graphics, creating animation, playing games and building 3D printing projects. The new OpenSUSE Leap version is available for download for both 32-bit and 64-bit systems. Existing OpenSUSE Leap users can upgrade their systems using the built-in update system.

Google blocks Android spyware family Lipizzan Google’s Android Security and Threat Analysis teams have jointly discovered a new spyware family that gets distributed through various channels including Play Store. Called Lipizzan, the software has been detected in 20 apps that have been downloaded on fewer than 100 devices. Unlike some of the earlier spyware, Lipizzan is a multi-stage spyware that can be used to monitor and exltrate email, text messages, location, voice calls and media. It


FOSSBYTES

is typically available as an innocuous-sounding app such as ‘Backup’ or ‘Cleaner’. Once installed, the spyware downloads and loads a second ‘license verication’ page that validates some abort criteria on the hardware. “If given the all-clear, the second stage would then root the device with known exploits and begin to exltrate device data to a command and control server,” the team, comprising Android Security’s Megan Ruthven and the Threat Analysis Group’s Ken Bodzak and Neel Mehta, have written in a blog post. The second stage of Lipizzan is capable of performing and exltrating results of tasks such as call recording, VoIP, voice recording, location monitoring, screenshot capturing and taking photos. Additionally, it is capable of helping attackers retrieve data from apps like Gmail, Hangouts, KakaoTalk, LinkedIn, Messenger, Skype, Snapchat, Viber and WhatsApp, among others. Google researchers found the presence of Lipizzan while investigating Chrysaor — a recently emerged spyware that was believed to be written by the NSO Group. Once spotted clearly, Google’s Play Protect service released a notication on all affected devices and removed the apps with Lipizzan from the online store. Moreover, Google has enhanced Play Protect’s capabilities to continuously detect and block targeted spyware on the Android platform. Developers need to use official resources only when building their apps to ensure a secured and safe experience.

GitHub adds new features to grow community engagements Supporting open source efforts by developers, GitHub has brought out a list of new features to enhance community engagements around your projects. “Thanks to some subtle (and not so subtle) improvements in the past few months, it’s now easier to make your rst contribution, launch a new project or grow your community on GitHub,” the GitHub team wrote in a blog post. First on the list of new features is contributor badges. Being a maintainer, you can now see a ‘rst-time contribution’ badge that helps you review pull requests from users who have contributed to your projects for the rst time. The ‘rst-time contributor’ badge becomes a ‘contributor’ badge in the comments section once the pull request is merged. Furthermore, you can expose the information in the additional ag via the GraphQL API. Apart from providing badges to your contributors, you have been provided with the option to add a licence le to your project using a new licence picker. This new section helps you pick an appropriate licence by providing the full text. It also allows you to customise any applicable elds prior to committing the le or opening a pull request. As privacy is one of the major factors preventing you from contributing to a new project, GitHub has added the ability to let you keep your email address private. GitHub also provides a warning that lets you make an informed decision about contributing to a project you were blocked from previously. Moreover, blocked users on the platform will not be able to comment on issues or pull requests in third-party repositories.

www.lulu.com

•

www.magzter.com

•

Createspace.com

•

www.readwhere.com

•


13

FOSSBYTES

Microsoft is now a part of the Cloud Native Computing Foundation Continuing its developments around open source, Microsoft has now joined the Cloud Native Computing Foundation (CNCF). The latest announcement comes days after the Redmond company entered the board of the Cloud Foundry Foundation. “Joining the Cloud Native Computing Foundation is another natural step on our open source journey, and we look forward to learning and engaging with the community on a deeper level as a CNCF member,” said Corey Sanders, partner director, Microsoft, in a joint statement. Microsoft has chosen the Platinum membership of the CNCF. Gabe Monroy, a lead product manager for containers on Microsoft Azure and former Deis CTO, is joining CNCF’s governing board. Led by the core team members of the Linux Foundation, CNCF has welcomed the new move of Microsoft. The non-prot organisation considers it a “testament to the importance and growth” of cloud technologies and believes the Windows maker’s commitment to open source infrastructure is a ‘signicant asset’ to its board. “We are honoured to have Microsoft, widely recognised as one of the most important enterprise technology and cloud providers in the world, join CNCF as a platinum member. Its membership, along with other global cloud providers that also belong to CNCF, is a testament to the importance and growth of cloud native technologies,” stated Dan Kohn, executive director of the Cloud Native Computing Foundation.

14

“We hope these improvements will help you make your rst contribution, start a new project, or grow your community,” GitHub concluded in its blog. First launched in October 2007, GitHub is so far used by more than 23 million people around the globe. The platform hosts over 63 million projects with a worldwide employee base of 668 people.

Mozilla aims to enhance AI developments with open source human voices While elite digital assistants like Alexa, Cortana, Google Assistant and Siri have so far been receiving inputs from users via the spoken word, Mozilla is planning to enhance all such existing articial intelligence (AI) developments by open sourcing human voices on a mass level. The Web giant has already launched a project called Common Voice to build a large-scale repository of voice recordings for future use. Mozilla has started capturing human voices since June to build its open source database. The database will be live later this year to “let anyone quickly and easily train voice-enabled apps” that go beyond Alexa, Google Assistant and Siri. “Experts think voice recognition applications represent the ‘next big thing’. The problem is the current ecosystem favours Big Tech and leaves out the next wave of innovators,” said Daniel Kessler, senior brand manager, Mozilla, in a recent blog post. Tech companies are presently using different voices to teach computers to understand the variety of languages for their solutions. But the data sets with the voice collections are mostly proprietary as of now. Therefore, a large number of developers have no access to voice recording samples to test their own voice recognition projects. This ultimately leads to a limited number of apps understanding our speech. Things are appearing to be changing with Common Voice. “The time has come for an open source data set that can change the game. The time is right for Project Common Voice,” Kessler stated. Mozilla is asking individuals to donate their voice recordings either on the Common Voice Web page or by downloading a dedicated iOS app. Once you are ready with your recording, you need to read a set of sentences that will be saved into the system. The recorded voices, which would come in a variety of languages with various accents and demographics, will be provided to third-party developers. In addition to simply receiving voice donations, Mozilla has built a model by which users will validate the recordings that are stored in the system. This process will help train an app’s speech-to-text conversion capabilities. All this will enable not just one or two but 10,000 hours of validated audio that will power tons of AI models in the near future. Notably, recordings received through the Common Voice initiative will be integrated into the Firefox browser as well. But the main purpose of this exercise is to provide a public resource.

For more news, visit www.opensourceforu.com


CODE

SPORT

Sandya Mannarswamy

In this month’s column, we discuss a few interview questions related to machine learning and data science.

A

s we have been doing over the last couple of months, we will continue to discuss a few more computer science interview questions in this column as well, particularly focusing on topics related to data science, machine learning and natural language processing. It is important to note that many of the questions are typically oriented towards practical implementation or deployment issues, rather than just concepts or theory. So it is important for interview candidates to make sure that they get adequate implementation experience with machine learning/ NLP projects before their interviews. Data science platforms such as Kaggle ( www.kaggle.com) host a number of competitions that candidates can attempt to practice their skills on. Also, many of the data science or machine learning related academic computer conferences host data challenge competitions such as the KDD Cup ( http://www.kdd.org/kdd-cup). Data science enthusiasts can sign on for these challenges and hone their skills in solving real life problems. Let us now discuss a few interview questions. 1. You are given 100,000 movie reviews that are labelled as positive or negative. You have been told to perform sentiment analysis on the new incoming reviews by classifying each review as positive or negative, which is a simple binary classication problem. Can you explain what features you would use for this classication problem? Once you decide on your set of features, how would you go about selecting which classier to use? 2. Let us assume that you decided to use the ‘bag of words’ approach in the above problem with each vocabulary term becoming a feature for your classier. Essentially, you can construct a feature set where the dimensions of this set are the same as the size of your vocabulary, and each feature corresponds to a specic term in the vocabulary. The feature value can either be the count of the term or merely the presence or absence of the

16


3.

4.

5.

6.

term in the document; or you can employ Tf-Idf count for each term-review combination, etc. You had used a random forests classier for sentiment classication. Now you are told that your vocabulary size is 100,000. Would this change your decision about which classier to use? For problem (1), you had decided to use a support vector machine classier. However, now you are told that instead of just doing binary classication of the reviews, you need to classify them as one of ve categories, namely: (a) strongly positive, (b) weakly positive, (c) neutral, (d) weakly negative, and (e) strongly negative. You are given labelled data with these ve categories now. Would you still continue to use the ‘support vector machine’ (SVM) classier? If so, can you explain how SVM handles multi-class classication? If you decide to switch from SVM to a different classier, explain the rationale behind your switch. For the sentiment classication problem, other than the review text itself, you are now given additional data about the movies. This additional data includes the reviewers’ names, address, age, country of residence, date of review and the specic movie genre they are interested in. This additional data contains both numeric and string data, with some of the features being categorical. A country’s name is string data, and the movie genre is string data which is actually categorical. What kind of data preprocessing would you do on this additional data to use it with your classier? Generally, interviewers expect you to be familiar with some of the popular libraries that can be used for data science. So some of the questions can be library-specic as well. In question (4), you may be asked to mention how you would convert categorical data to numeric form. Can you write a piece of Python code to do this conversion? Let us assume that you decided to use a SVM

Guest Column

classier for the sentiment classication problem. You nd that your classier takes a long time to t the training data. How would you reduce the training time? List all the possible approaches. 7. One of our readers suggested feature scaling/data normalisation as a preprocessing step before you train your model always. Is she correct? Is feature scaling or normalisation always needed in all types of classiers? Why do you think feature scaling can help achieve faster convergence of your learning procedure? One of the wellknown methods of feature scaling is Min-Max scaling. By feature scaling, you are actually throwing away your knowledge of the maximum and minimum values that the feature can take. Wouldn’t the loss of this information affect the accuracy of your classier on unseen data? If you are using a decision tree classier or random forests, should you still do feature scaling? If yes, explain why. 8. Scikit-learn is a popular machine learning library available in Python, which provides ready-made implementations of several classiers such as decision tree, support vector machine, random forests, logistic regression, multilayer perceptron, etc. These classiers provide a ‘predict’ function, which predicts the output for a given data instance. They also provide a ‘predict_proba’ function, which returns the probability for each sample (data instance) belonging to a specic output class. For instance, in the case of the movie review sentiment prediction task, with two classes positive and negative, the ‘predict_proba’ function would return the probability of the sample belonging to the positive sentiment category and negative sentiment category. When would you use the ‘predict_proba’ function in your sentiment classication task? 9. In the sentiment classication problem on the movie reviews data, you found that some of the reviews did not have the date, country of reviewer and the movie genre. How would you handle these missing data? Note that these features were not numeric; so what kind of data imputation would make sense in this case? 10. In the movie reviews training labelled data set, you are now given certain additional data features that include: (a) the star rating reviewers give to the movie, (b) whether they would like to watch it again, and (c) whether they liked the movie. Would you use these additional features in your training data to train your model? If not, explain why you wouldn’t. 11. What is the data leakage problem in machine learning and how do you avoid it? Does the scenario mentioned in question (10) fall under the data leakage category? Detailed information on data leakage and its avoidance can be found in this well-written and must-read paper ‘Leakage in data mining: formulation, detection, and avoidance’ which was presented at the KDD 2011 conference and is available at http://dl.acm.org/citation.cfm?id=2020496. 12. You are using k-fold cross validation for selecting the hyperparameters of your model. Given that your training data has

CodeSport

features which are on widely varying scales, you have decided to do feature scaling. Should you do data scaling once for the entire training data set and then perform the k-fold cross validation? Or should you do the feature scaling within each fold of cross-validation? Explain the reason behind your choice. 13. You are given a data set which has a large number of features. You are told that only a handful of these features are relevant in predicting the output variable. Will you use Lasso regression or ridge regression in this case? Explain the rationale behind your choice. As a follow-up question, when would you prefer ridge regression over Lasso? 14. Decision tree classiers are very popular in supervised machine learning problems. Two well-known tree classiers are random forests and gradient boosted decision trees. Can you explain the difference between the two of them? As a follow-up question, can you explain ensemble learning methods in general? When would you opt for an ensemble classier over a non-ensemble classier? 15. You are given a data set in which many of the variables are categorical string variables. You decided to encode the categorical variables with One Hot Encoding. Consider that you have a variable called ‘country’, which can take any of the 20 values. With One Hot encoding, you end up creating 20 new feature variables in place of the single ‘country’ variable. On the other hand, if you use label encoding, you convert the categorical string variable to a categorical numerical variable. Which of the two methods leads to the ‘curse of the dimensionality’ problem? When would you prefer to go for One Hot encoding vs label encoding? Please do send me your answers to the above questions. I will discuss the solutions to these questions in next month’s column. I also wanted to alert readers about a new deep learning specialisation course by Prof. Andrew Ng coming up soon on the Coursera platform ( https://www.coursera. org/specializations/deep-learning). If you are interested in becoming familiar with deep learning, there is no better teacher than Prof. Ng whose machine learning course on Coursera is being taken by more than a million students. If you have any favourite programming questions/software topics that you would like to discuss on this forum, please send them to me, along with your solutions and feedback, at sandyasm_ AT_yahoo_DOT_com. Till we meet again next month, wishing all our readers wonderful and productive days ahead.

By: Sandya Mannarswamy The author is an expert in systems software and is currently working as a research scientist at Conduent Labs India (formerly Xerox India Research Centre). Her interests include compilers, programming languages, le systems and natural language processing. If you are preparing for systems software interviews, you may nd it useful to visit Sandya’s LinkedIn group ‘Computer Science Interview Training India’ at http://www.linkedin.com/ groups?home=&gid=2339182


17

NEW PRODUCTS Pocket-friendly Bluetooth speaker from Kodak Super Plastronics Pvt Ltd, a brand licensee of Kodak, has launched its rst Bluetooth portable speaker in India, the Kodak 68M. The device offers a complete sound experience in an affordable price, company sources claim. Apart from Bluetooth connectivity, the speaker supports an auxiliary wire and a micro USB jack. It is equipped with a low sound output with a reach of up to 10 metres. Powered by a 3.7W battery, the speaker is capable of lasting for over ve hours. For enhancing the sound experience, the device can also be connected to an additional speaker. The Kodak speaker can also be connected to any TV, with or without Bluetooth, making it a complete package for entertainment lovers. The Kodak 68M speaker is available online and at retail stores.

Price:

` 3,290 Address: Super

Plastronics Pvt Ltd, 1st Floor, Dani Corporate Park, 158 Dani Compound, Vidya Nagari Road, Kalina, Santacruz East, Mumbai – 400098; Ph: 022-66416300

18

Earbuds with heart-rate monitor from Jabra Audio and connectivity devices manufacturer, Jabra, has introduced superior quality earbuds for music and voice calls, called Jabra Elite Sport. The device sports advanced wireless connectivity, which lters out background noise ensuring distraction-free usage. The buds come with the ease of portable charging and deliver 4.5 hours of playtime. They have customisable tting options, enabling users to stay connected comfortably during outdoor and sports activities. A special button on the earbuds, ‘Hear Through’, lters out surrounding noise. The device has four microphones and offers personalised tness analysis using an in-ear heart rate monitor.

Price:

` 18,999 With IP67 certification, the earbuds have a three-year warranty for damage by sweat and water, enabling hasslefree usage. The wireless Jabra Elite Sport is available in lime green, grey and black, online and at retail stores. Jabra India Pvt Ltd, Redington India Limited, New No. 27, NRS Building, Velachery Road, Saidapet, Chennai-600015. Address:

Mechanical keyboard for gamers from Galax Galax, the manufacturer of gaming products, has unveiled its latest HOF Black edition mechanical keyboard, specially designed for gamers. The keyboard uses a genuine Cherry MX mechanical key switch with 50 million keystrokes for long lasting and quick response, giving users a stable and long-term option. The stylish-looking keyboard is built with an anodised (black)/baking paint (white colour) aluminium plate. It offers up to 112 lighting effects with software and 88 lighting effects without software. Its Macro keys make each key of the device programmable. The keyboard comes with media control buttons along with a die-cast volume and lighting roller. With the n-key rollover, the company claims the keyboard is 100 per cent antighosting. The HOF keyboard can enter all the signals accurately, even when played faster or when pressing multiple keys. It has a USB 2.0 hub with audio-out


Price:

` 7000 and a mic-in jack. The hub allows the users to connect USB devices of all types quickly. It also has a magnetic, detachable, soft-touch wrist rest to make prolonged use comfortable. The Xtreme Tuner Plus system enables users to customise the keyboard by controlling Macros. It also has per-key programming, back light setting and lighting patterns. The HOF black edition carries a three-year warranty period and is available at Amazon. Address: Amazon

India, Brigade Gateway, 8th Floor, 26/1, Dr Rajkumar Road, Malleshwaram West, Bengaluru, Karnataka – 560055; Ph: 1800-30009009

Budget-friendly smartphone with front LED flash from Lenovo Lenovo has launched an affordable dual camera smartphone, the Lenovo K8 Note. The device sports a 13.9cm (5.5 inch) full HD (1080 x1920 pixels) display with Corning Gorilla Glass protection. It is powered by a decacore MediaTek MT6797 SoC with four Cortex-A53 cores clocked at 1.4GHz and 1.85GHz, as well as two Cortex-A72 cores clocked at 2.3GHz. Built with 5000 series aluminium and polycarbonate, the device is supposed to be splash-resistant. The dual SIM (Nano) device runs on Android 7.1.1 Nougat and is backed with a huge 4000mAh battery with turbo charging. On the camera front, the smartphone sports a rear 13 megapixel primary sensor, accompanied by a 5

megapixel depth sensor with a dual LED-CCT ash module; and a 13 megapixel front camera with an LED ash module for seles. The connectivity options of the device include 4G VoLTE, dual band (2.4GHz and 5GHz), Wi-Fi 802.11ac, Bluetooth v4.1, GPS, micro USB and a 3.5mm audio jack. The Lenovo K8 Note comes in two variants – 3GB RAM/32GB storage and 4GB RAM/64GB storage — both available in ‘Fine Gold’ and ‘Venom Black’ colours online and at retail stores. Price:

` 12,999 for the 3GB Lenovo India Pvt Ltd, Vatika Business Park, 1st Floor, Badshah Pur Road, Sector-49, Sohna Road, Gurugram-122001 Address:

RAM/32GB storage option and ` 13,999 for the 4GB RAM/64GB storage variant.

Water-resistant Bluetooth headphones Motorola has introduced its Bluetooth in-ear headphones in India – the Verve Loop. Aimed at sports and tness enthusiasts, the headphones offer a hassle-free, comfortable t during outdoor and workout sessions. The device comes with an IP54 rating for water and splash resistance, enabling damage-free use. It is designed to deliver a balanced, high quality audio experience along with easy Bluetooth pairing with voice prompts. Powered by a lithium-ion battery, the device delivers up to six hours of playback with a single charge. Company sources also claim that it provides one hour of play time on just a 20-minute charge. The headphones offer balanced sound at any volume and superb noise isolation. Features include A2DP (Advanced Audio Distribution Prole), HFP (Hands Free Protocol) and AVRCP (Audio/Video Remote

from

Motorola

Control Prole), enabling hands-free calling and voice assistance. The headphones come with inline control buttons for volume, play/ pause, etc, three sets of extra ear gels and three sets of ear hooks for stable support. The Motorola Verve Loop is compatible with all Android and Apple smartphones and tablets, apart from supporting Siri and Google Now voice assistants. Available in combinations of charcoal grey/black and orange/ black, the headphones can be purchased online and at retail stores. Price:

` 2,499 Address: Motorola Solutions India, 415/2, Mehrauli-Gurugram Road, Sector 14, Near Maharana Pratap Chowk, Gurugram, Haryana – 122001; Ph: 0124-4192000; Website: www.motorola.in

The prices, features and specications are based on information provided to us, or as available on various websites and portals. OSFY cannot vouch for their accuracy.

Compiled by: Aashima Sharma


19

Exploring Software

Guest Column

Anil Se th

Importing GNUCash Accounts in GNUKhata gkcore is the REST API core engine of GNUKhata. The GNUKhata app comprises two applications — gkcore and gkwebapp. The objective of this tutorial is to get to know the API.

G

NUKhata is an application developed using the Pyramid Web Framework. It comprises two Web applications – a core application called gkcore, and a Web application called gkwebapp. You may easily get started with the installation and development by referring to https:// gitlab.com/gnukhata/gkwebapp/wikis/home. As a way of learning how to extend GNUKhata, you may consider importing data from GNUCash into GNUKhata. Since the core and user interface are two separate applications, a good way to learn the core application interface is to create a utility program which will add the GNUCash data. The utility program will rst need to log into the core server, and then issue the commands to add the needed data. Make sure that you are able to run the core and the Web applications; use the latter to create an organisation and an admin user for the organisation. It is important to keep in mind that the gkcore application needs to be run using the gkadmin user, assuming that you are following the steps from the wiki article; otherwise, it will not be able to access the database.

The login process You may examine gkwebapp/views/startup.py to understand the logic of the steps needed for logging in. The process involves selecting an organisation rst, and then supplying the credentials of a user for that organisation. In order to keep the code as simple as possible, as the objective is to learn the API, select the rst organisation. The login credentials are hard-coded. In case of any errors, the utility will just crash and not attempt any error handling. Once the login is successful, a token is issued. This token will authorise all subsequent calls to the core server. You will notice that the calls to the core server are simple get or post requests. The data objects transferred between the two are JSON objects. import requests, json

return requests.post(gkhost + route,data=jsondata,headers =hdrs).json() def getOrg(): gkdata = getJsonResponse(‘organisations’)[‘gkdata’] rst_org = gkdata[0] route= ‘/’.join([‘orgyears’,rst_org[‘orgname’], rst_ org[‘orgtype’]]) gkdata = getJsonResponse(route)[‘gkdata’] return gkdata[0][‘orgcode’] def orgLogin(orgcode): gkdata = {‘username’:’anil’, ‘userpassword’:’pswd’,’orgc ode’:orgcode} return postJsonResponse(‘login’,json.dumps(gkdata)) [‘token’] orgcode = getOrg() gktoken = orgLogin(orgcode)

Adding accounts GNUCash can export the accounts and transactions in CSV les. In the current article, you may extract the accounts into a le, accounts.csv. The Python CSV modules make it very easy to handle a CSV le. The rst row contains the column labels and should be ignored. You may use the DictReader for more complex processing of the le. For this application, in which only a few columns are needed, the CSV reader is adequate. There are a few differences in the top level account/group names of GNUCash and GNUKhata. So, you need to create a dictionary to map the names from GNUCash to the ones used in GNUKhata. Some groups in the level below ‘Assets’ in GNUCash appear as top level groups in GNUKhata, e.g., ‘Current Assets’ and ‘Fixed Assets’. You may ignore ‘Assets’ from the account hierarchy when transferring the data. As before, the code below ignores error handling and assumes ‘all is well’:

gkhost = ‘http://127.0.0.1:6543/’ def getJsonResponse(route,hdrs=None): return requests.get(gkhost + route, headers = hdrs).json()

import csv def addSubGroup(name,parent,header): data = json.dumps({‘groupname’:name,

def postJsonResponse(route,jsondata,hdrs=None):

20

‘subgroupof’:parent})


Exploring Software

Guest Column

return postJsonResponse(‘groupsubgroups’,data,header) [‘gkresult’]

login_hdr = {“gktoken”:gktoken} toplevel_mapping = { ‘Capital’:’Capital’, ‘Current Assets’:’Current Assets’,

def addAccount(name,parent,header):

‘Fixed Assets’:’Fixed Assets’,

data = json.dumps({‘accountname’:name, ‘groupcode’:parent

‘Liabilities’:’Current Liabilities’,

,’openingbal’:0.00})

‘Expenses’:’Direct Expense’,

res = postJsonResponse(‘accounts’,data,header)

‘Income’:’Direct Income’} createAccounts(‘accounts.csv’, toplevel_mapping, login_hdr)

def createAccounts(fn,toplevel,header): # get exiting groups and their codes groups = getJsonResponse(‘groupsubgroups?groupatlist’,he ader)[‘gkresult’] f = open(fn) rows = csv.reader(f) skip_rst = rows.next() for row in rows: fullname = row[1].split(‘:’) name = row[2] is_group = row[-1] == ‘T’ # Map top-level to GNUKhata top level # Ignore Assets and use ‘Current Assets’ and ‘Fixed Assets’ as top level if fullname[0] == ‘Assets’: fullname = fullname[1:] if len(fullname) > 1 and fullname[0] in toplevel:

fullname[0]=toplevel[fullname[0]] parent = groups[fullname[-2]] if is_group:

There are some potential issues in transferring accounts from GNUCash to GNUKhata. For example, in GNUKhata, you either have an account or a sub-group. However, in GNUCash, a sub-group can function as a normal account as well. An account in GNUCash is consistent with a sub-group of GNUKhata if the placeholder ag is true. However, the objective of this article is to become familiar with the communication between a client application and the core server, and no attempt is made to handle corner cases. GNUCash transaction data may also be exported as CSV les. The above utility may be similarly extended to handle that data as well. The Web-based frontend, gkwebapp, makes it easy to view and enter the data. The communication with the server happens as in the utility above, and it is done from the code residing in the views directory of the gkwebapp. You may, as an exercise, extend the Web application to import GNUCash accounts into GNUKhata and learn that as well!

if not (name in groups): # add this to the list of groups

By: Dr Anil Seth

groups[name]=addSubGroup(name,

The author has earned the right to do what interests him. You can fnd him online at http://sethanil.com, http://sethanil. blogspot.com, and reach him via email at [email protected].

parent,header)

else:

addAccount(name,parent,header)

OSFY Magazine Attractions During 2017-18 MONTH

THEME

March 2017

Open Source Firewall, Network security and Monitoring

April 2017

Databases management and Optimisation

May 2017

Open Source Programming (Languages and tools)

June 2017

Open Source and IoT

July 2017

Mobile App Development and Optimisation

August 2017

Docker and Containers

September 2017

Web and desktop app Development

October 2017

Artificial Intelligence, Deep learning and Machine Learning

November 2017

Open Source on Windows

December 2017

BigData, Hadoop, PaaS, SaaS, Iaas and Cloud

January 2018

Data Security, Storage and Backup

February 2018

Best in the world of Open Source (Tools and Services)

22 | SEPTEMBER 2017 | OPEN SOURCE FOR YOU | www.OpenSourceForU.com

For U & Me Interview

is built on a lot of past experience Q

How did you start your journey with the CentOS project? It was in late 2004. I was not one of the founders of CentOS but showed up on the scene in its early days. At that time, we had a small team and a lot of machines running Red Hat Linux 7.3 and Red Hat Linux 9. With Red Hat moving down the path towards Red Hat Enterprise Linux, a model that didn’t work well for us, I started looking at options. We explored Debian and SUSE initially, but found management and lifecycle on each of them hard to map out workow into. It was during this time that I came across the Whitebox Linux effort and then the CentOS Project. Both had the same goal, but the CentOS team was more inclusive and seemed more focused on its goals. So, in late September 2004, I joined the CentOS IRC channel and then, in November, I joined the CentOS mailing list as a contributor. And I am still contributing 13 years down the road!

Q

What were the biggest roadblocks that emerged initially while designing CentOS for the community, and how did its core development team overcome them? A lot of our problems were about not getting off the ground. Initially, there was no clear aim. And then, we faced the challenge that the build systems and code audit tools in 2003/2004 were either primitive, absent entirely or the contributors were unaware of them. A Linux distribution is a large collection

24

Managing a Linux distribution for a long time requires immense community effort. But what is the key to success in a market that includes hundreds of competitive options? Also, what are the challenges in building a brand around an open source offering? Karanbir Singh, project leader, CentOS, answers these questions and outlines the future of the platform that has been leading Web developments, in an exclusive conversation with Jagmeet Singh of OSFY.

of a lot of code, written in many languages — each with its own licence, build process and management. Three main strategies saw us past that painful process. The rst was consistency. Whatever we did, we had to be consistent and uniform across the entire distribution, and make sure all developers had a uniform understanding of the process and ow. The second was a self-use focus. Regardless of what the other people were targeting, all developers were encouraged to focus on their own use cases and their personal goals. The third was the hardest, to try and disconnect commercial interests from developer and contributor work.


Q

Why was there a need for CentOS Linux when Fedora and Red Hat Enterprise Linux already existed in the market? The Fedora project was still getting sorted out around then. Its team had a clear mandate to try and build an upstream-friendly Linux distribution that was going to move fast and help the overall Linux ecosystem mature. Red Hat Enterprise Linux, on the other hand, has been built for the commercial medium to large organisations, looking for value above the code. This left a clear gap in the ecosystem for a community-centric, manageable, predictable enough Linux distribution that the community itself, small vendors, and niche users around the mainstream could consume. Initially, the work we did was quite focused on the specic use cases that the developers and contributors had. All of us were doing specic things, in specic ways and CentOS Linux tted in well. But as we started to mature, we saw great success in specic verticals, starting from academia and education institutions to Web hosting, VoIP (Voice over Internet Protocol) and HPC (high performance computing).

Q

What were the major outcomes of the Red Hat tie-up? Red Hat came on as a major sponsor for the CentOS Project in January 2014. From the CentOS Project’s perspective, this meant we were then able to start looking beyond just the platform and the Linux distribution. It allowed us to build the infrastructure and resources needed to support

Interview

For U & Me

Karanbir Singh, project leader, CentOS

other projects above the platform, develop a great CI (continuous integration) service as well as a much better outreach effort than we were able to earlier. The real wins have been from the user perspective. If today you are looking for an OpenStack user side install, the CentOS Project hosted on the RDO stack is the best packaged, tested and maintained option. In a nutshell, the Red Hat relationship has allowed the CentOS Project to dramatically expand the scope of its operations beyond just the Linux distribution and enable many more user scenarios.

doing so in order to achieve a goal— either to run a website or to run a mail server. Helping users achieve their end goals easily has been our constant focus. It is a key metric we still track in order to reach our goals. This means that as the user base adapts to the new world of cloud-native, container-based and dynamically orchestrated workloads, the CentOS Project continues to deliver the same level of user focus that we have had over the years. Protecting the user’s investment in the platform across the base without giving up on existing processes is something we deliver till date. For instance, people can choose when and how they jump on the container process, or just entirely opt out. It is not something that will be inuenced by a CentOS Linux release. It is this duality, which maintains an existing base while allowing the user to seamlessly move into emerging tech, that creates a great value proposition for CentOS Linux.

Q

How do you manage the diversication of different CentOS Linux versions and releases?

Q

What makes CentOS Linux a perfect choice even 13 years after its rst release in May 2004? When users install a Linux distribution, they are almost always

The way the contribution process and ownership works makes it relatively easy to manage the diversication. Primarily, the aim is to ensure that if we are doing something specic, the people doing the work are directly invested in the result of the work itself. This helps ensure quality as there are eyes scrutinising incoming patches and changes—since the developers’ own requirements could be impacted by shipping a sub-optimal release.

www.OpenSourceForU.com | OPEN SOURCE FOR YOU | SEPTMBER 2017 |

25


Participation from the target audience for the specic media or release is a very critical requirement. And because this typically comes through the common project resources, it also means that the people doing this work are well engaged in the core project scope and Linux distribution areas, allowing them to bridge the two sides nicely. At the moment, there are dozens of different kinds of CentOS releases, including atomic hosts, minimal installs, DVD ISOs, cloud images, vagrant images and containers. Each of these comes through a group that is well invested in the specic space.

contributors don’t always have the time to work on each request, but if you look at the CentOS Forums, almost every question gets a great result. There is also a lot of diversity in the groups. The CentOS IRC channel idles at over 600 users during the day, but a large number of users never visit the forums. Similarly, the CentOS mailing lists include over 50,000 people, but a large number of them are never reaching the IRC channel or the forums.

Q

What are the major differences you’ve observed being a part of a corporate entity like Red Hat and a community member? Which is tougher among the two?

Q

What are the various efforts that lead to a consistent developer experience on CentOS Linux? Application developers working to consume CentOS Linux as their base can trust the API/ABI efforts where content deployed today will still work three or ve years down the road. The interfaces into that don’t change (they can evolve, but won’t break existing content/scripts/code). Therefore, working with these interfaces also means that they work within the same process that the end user is already aware of and already manages for simple things like security patching, an area often overlooked by the casual developer.

Q

How does the small team of developers manage to offer a long period of support for every single release? We invest heavily in automation to enable long-term support. And that means that a very small group of people can actually look after a very large codebase. It changes the scope of what the contributors need to do. Rather than working on the code process, we work on the automation around it, and aggressively test and monitor the process and code. The other thing is that we get community support. Developers and

26

most of my focus is around enablement and making sure that contributors and developers have the resources they need to succeed. The other 70 per cent of my time is spent as a consulting engineer at Red Hat, working with service teams, helping build best practices in operations roles and modern system patterns for online services. Additionally, I have been involved in some of the work going on in the containers world and its user stores, which includes DevOps, SRE-Patterns, CI and CD, among others.

Q

Is it solely the community feedback that helps you build new CentOS updates, or do you also consider feature requests from the Red Hat team?

CentOS Linux is built as a downstream from Red Hat’s sources. They are being delivered via git.centos.org, and then the additional layers and packages are built by various community members. We also encourage people to come and join the development process to build, test and deliver features to the audience that we target through the open source project. All this is entirely community focused. So if someone at Red Hat wants to execute something, they would need to join the relevant community and work that route for engagement on CentOS. Having said that, we have a concept of Special Interest Groups that can be started by anyone, with a specic target in mind. Of course, this is only above the Linux distribution itself.

Q

Apart from your engagements related to being a CentOS Project member, what are your major tasks at Red Hat? These days, I spend around 30 per cent of my time working on the CentOS Project. Rather than in the project itself,


I have been involved with open source communities for over 15 years. During such a long period, open source work has never been my primary job. It’s always been something that I do in my free time or in addition to what I was already doing, similar to my move with the CentOS Project. But what makes Red Hat unique in a way is that this isn’t an odd role. A large number of people at Red Hat participate and execute their day job via open source communities. And that makes it a lot easier, being a longterm contributor. There is only one key challenge that one needs to keep in mind when working on an open source project as a part of the day job, though. It is to set realistic expectations around community participation, and recognise that the community is there because its members often care about something far more than the people paid to work on a project. However, this typically isn’t a concern when a community comes together around smaller pieces of the code. The CentOS Project has quite a widespread and extensive footprint. It involves talking to and participating in a wider community where a large majority is unknown personally. Managing expectations and ensuring


there is respect both ways has been a balancing act that’s never easy. It’s something I work quite hard on, and hope we can keep getting it better.

Q

What is your advice to those who are planning to begin with CentOS? The most important thing to remember for those planning to begin with CentOS is that you are not alone. There is lots of documentation as well as tremendous support and help from a vast community. So as a new user, make sure you engage with the existing user base, ask questions, and spend a bit of time to understand what and how things are. In the long term, understanding the details will pay off. CentOS Linux is built on a lot of past experience. Anyone starting down the path of adoption should keep this in mind. We’ve tried to build a culture of helping those that need the most help, but also encourage new users to learn and grow with the community.

28

Q

What are the biggest features that we can see in the next CentOS release? The Special Interest Groups (SIGs) are constantly releasing new content all the time. As an example, the Cloud SIG released OpenStack Ocata within a few hours of the upstream project release. There is also a lot of work being done in the Atomic SIG around containers, and in the Virt SIG on existing and upcoming virtualisation technologies.

Q

Lastly, where do you see CentOS in the future of open source? CentOS Linux, due to its characteristics, is a great t for areas like Web hosting, Web services, cloud workloads and container delivery. Also, as a platform for long-term community centric workloads, it is a good option in areas like HPC and IoT. The Linux distribution also specically suits the needs of the education sector—starting and supporting not only IT education


but education as a whole. Moreover, I would like to see CentOS Linux extend its footprint as a vehicle for enablement. And this is where the CentOS Project provides a great space for other open source projects—building services for CI/CD, interacting with the users, nding the niches that people care about and solving real-world problems. If you are involved today with an open source project, I strongly encourage you to get in touch with me and discuss its development areas. We measure our success on the basis of how successful CentOS has been for the people, the communities, the open source projects and the users who have invested their time, resources and support for the CentOS Project. And we look forward to solving more problems, building better solutions and bridging more gaps together. You can reach Karanbir directly at [email protected] or meet him on Twitter at @kbsingh

Let’s Try Admin

A Primer on Software Defined Networking (SDN) and the OpenFlow Standard Continuous innovation and the need to adapt to the constraints of conventional networking has made software defined networking (SDN) pretty popular. It is an approach that disassembles the network’s control plane and data plane. This allows network administrators to program directly without having to worry about the hardware specifications.

penFlow, the rst SDN standard, is a communication protocol in software dened networking (SDN). It is managed by the Open Networking Foundation (ONF). The SDN controller or the ‘brain’ interacts with the forwarding (data) plane of the networking devices like routers and switches via OpenFlow APIs. It empowers the network controllers to decide the path of network packets over a network of switches. The OpenFlow protocol is required to move network control out of exclusive network switches and into control programming that is open source and privately overseen. Software-dened networking uses southbound APIs and northbound APIs. The former are used to hand over information to the switches and routers. OpenFlow is the rst southbound API. Applications use the northbound APIs to interact.

O

simulator with SDN technology. An OpenFlow switch is a package that routes packets in the SDN environment. The data plane is referred to as the switch and the control plane is referred to as the controller. The OpenFlow switch interacts with the controller and the switch is managed by the controller via the OpenFlow protocol. The fundamental components of the OpenFlow switch (as shown in Figure 2) incorporate at least one flow table, a meter table, a group table and an OpenFlow channel to an exterior controller. The flow tables and group table perform the packet scanning and forwarding function based on the flow entries configured by the controller. The routing decisions made by the controller are deployed in the switch’s flow table. The meter table is used for the measurement and control of the rate of packets.

Porting an OpenFlow switch in ns-3

Configuring the SDN OFSwitch

The OpenFlow 1.3 module for ns-3, widely known as the OFSwitch13 module, was intended to boost the ns-3 network

In this article, we have incorporated the OFSwitch13 (version 1.3) with ns-3. To benet from the features of OFSwitch13, an www.OpenSourceForU.com | OPEN SOURCE FOR YOU | SEPTEMBER 2017 |

29

Admin

Let’s Try

Traditional Network Architecture

SDN Architecture

SDN Controller

Centralized Control Plane

Distributed Control Plane

OpenFlow App

App

OFSwitch

Internal Diagram of OF Switch

Meter Table

Group Table

Secure Channel Software

Flow Table

Flow Table

Flow Table

Hardware

- Data Plane - Control Plane

Figure 1: Traditional network architecture vs SDN architecture

ofsoftswitch13 library is used. All the commands given below have been tested on Ubuntu 16.04 and might change for other versions or distributions. Before that, there are a few bundles to be introduced on the system:

Hosts

Figure 2: OpenFlow switch components

In the ns-3.26 directory, download the repository of OFSwitch13, as follows:

$ sudo apt-get install build-essential gcc g++ python git mercurial unzip cmake

$ hg clone https://bitbucket.org/ljerezchaves/ofswitch13-

$ sudo apt-get install libpcap-dev libxerces-c-dev libpcre3-

module src/ofswitch13

dev ex bison

$ cd src/ofswitch13

$ sudo apt-get install pkg-cong autoconf libtool libboost-

$ hg update 3.1.0

dev

$ cd ../.. $ patch -p1 < src/ofswitch13/utils/ofswitch13-src-3_26.patch

In order to utilise ofsoftswitch13 as a static library, you need to introduce the Netbee library, as the ofsoftswitch13 library code relies upon it. $ wget https://bitbucket.org/ljerezchaves/ofswitch13-module/ downloads/nbeesrc.zip $ unzip nbeesrc.zip (for unzipping) $ cd netbee/src/

$ patch -p1 < src/ofswitch13/utils/ofswitch13-doc-3_26.patch

The le ofswitch13-src-3_26.patch will allow OFSwitch to get raw packets from nodes (devices). To do this, it will create a new OpenFlow receive callback at CsmaNetDevice and virtualNetDevice. The le ofswitch13-doc-3_26.patch is optional but preferable. After successful installation, congure the module, as follows:

$ cmake . $ make

$ ./waf congure --with-ofswitch13=path/to/ofsoftswitch13

$ sudo cp ../bin/libn*.so /usr/local/lib

$ ./waf congure --enable-examples --enable-tests

$ sudo ldcong $ sudo cp -R ../include/* /usr/include/

Now, clone the repository of the ofsoftwsitch13 library, as follows: $ git clone https://github.com/ljerezchaves/ofsoftswitch13 $ cd ofsoftswitch13

Now, we’re all set. Just build the simulator using the following command: $ ./waf

Enjoy the ns3.26 simulator with the power of SDN, i.e., OFSwitch 1.3.

$ ./boot.sh $ ./congure --enable-ns3-lib $ make

Integrating OFSwitch with ns-3 To install ns-3.26, use the following command: $ hg clone http://code.nsnam.org/ns-3.26

30

Simulating the basic network topology with SDN based OFSwitch In this section of the article, we’ll simulate a basic network topology with three hosts, a switch and a controller. Figure 3 demonstrates the topology of the network that we want to create. It includes three hosts, one switch and one controller.


Let’s Try Admin Here, host2 pings the other two Here, ofswitch13 domain comes into hosts—host1 and host3. Whenever either action. of the hosts makes a ping request, it is forwarded to the switch. This is indicated Ptr of13Helper=Cr by arrows shown in blue. As this is the eateObject(); first request, the switch’s flow table will of13Helper->InstallController not contain any entry. This is known as (controllerNode); a table miss. Thus, the request will be //to install forwarded to the controller. controller on node The controller will instruct the switch of13Helper->InstallSwitch (switchNode, with respect to the routing decision to be switchPorts); Figure 3: Network topology made and to also modify the ow table in //to install OFSwitch the switch. This is shown by the arrows in of13Helper->CreateOpenFlowChannels (); green. The request will then be forwarded by the switch to the appropriate destination, i.e., to host1 and host3. //for creating channels between Switch and The next time the same request is forwarded to the switch, controller the switch’s ow table will contain an entry for that request and, thus, the switch itself will make routing decisions based on Ipv4AddressHelper ipv4helpr; //set IPv4 addresses that entry, without the controller in action. Ipv4InterfaceContainer hostIpIfaces; The explanation for the code to simulate the above ipv4helpr.SetBase (“10.97.7.0”, “255.255.255.0”); topology is given below. Only the required extracts of //IPv4 range starts from the code are given. The accompanying lines of code 10.97.7.0 demonstrate the extra header documents required for rehostIpIfaces = ipv4helpr.Assign (hostDevices); enacting wired systems. Below lines will congure ping applications between hostsController

Switch

Host 1

Host 3

Host 2

#include #include

V4PingHelper pingHelper = V4PingHelper (hostIpIfaces. GetAddress (1));

The following lines of code create an object called ‘hosts’ of class NodeContainer common to all the other nodes. Here, three hosts are created.

pingHelper.SetAttribute (“Verbose”, BooleanValue (true)); ApplicationContainer pingApps1= pingHelper.Install (hosts. Get (0)); pingApps1.Start (Seconds (1));

NodeContainer hosts;

ApplicationContainer pingApps2 = pingHelper.Install (hosts.

hosts.Create (3);

Get (0)); pingApps2.Start (Seconds (1));

Ptr switchNode = CreateObject ();

Here, two hosts and one object of class ApplicationContainer called pingApps is created. Now the code for simulator to work-

//to create node for switch CsmaHelper csmaHelper; NetDeviceContainer hostDevices; NetDeviceContainer switchPorts;

Simulator::Stop (Seconds (10));

for(size_t i = 0;i
//simulation time is 10 seconds

Simulator::Run (); Simulator::Destroy ();

NodeContainer pair (hosts.Get (i), switchNode); NetDeviceContainer link = csmaHelper.Install (pair); hostDevices.Add (link.Get (0)); //two way linking switchPorts.Add (link.Get (1));

It is recommended that you save ofswitch13-modify.cc at this path (ns-dev/scratch/ ). To run the program, use the following command:

} $ ./waf --run ofswitch13-modify Ptr controllerNode = CreateObject (); //to create node for controller

The output is given in Figure 4. As shown in the gure, the host with the IP address 10.97.7.2 pings the other two hosts, and the ping is www.OpenSourceForU.com | OPEN SOURCE FOR YOU | SEPTEMBER 2017 |

31

L o o n y c o r n Some of our recent courses: On Udemy: 

GCP: Complete Google Data Engineer and Cloud Architect Guide



Time Capsule: Trends in Tech, Product Strategy



Show And Tell: Sikuli - Pattern-Matching and Automation

On Pluralsight: 

Understanding the Foundations of TensorFlow



Working with Graph Algorithms in Python



Building Regression Models in TensorFlow

L o o n y c o r n Our Content: 















The Careers in Computer Science Bundle 9 courses | 139 hours The Complete Machine Learning Bundle 10 courses | 63 hours The Complete Computer Science Bundle 8 courses | 78 hours The Big Data Bundle 9 courses | 64 hours The Complete Web Programming Bundle 8 courses | 61 hours The Complete Finance & Economics Bundle 9 courses | 56 hours The Scientific Essentials Bundle 7 courses | 41 hours ~30 courses on Pluralsight ~80 on StackSocial ~75 on Udemy

About Us:  

ex-Google | Stanford | INSEAD 80,000+ students

Admin

Let’s Try

Figure 4: Output of ofswitch13-modify.cc Figure 6: Position of hosts in NetAnim

Figure 5: Output of the log file

successful with nine packets being transmitted. The time statistics are also shown. In the program’s code, to view the log le, set trace=true. This will generate switch-stats-1.log in the ns-dev folder.

Visualising the basic working of the controller switch using Net Animator Network Animator, also known as NetAnim, is used to graphically portray projects in ns-3. It is an ofine animator, which animates the XML le generated during the simulation program in ns-3. ns-2 has many default animators for use but ns-3 is furnished with no default animator. So, we have to integrate NetAnim with ns-3. NetAnim version 3.107 is used for visualising. To visualise the above program code ( ofswitch13-modify. cc) in NetAnim, take the following steps. Add the following few lines of code:

Figure 7: Packet flow in NetAnim

The other three nodes are the created hosts. Node 0: 10.97.7.1 Node 1: 10.97.7.2 Node 2: 10.97.7.3 Figure 7 is a screenshot of the generated XML le with graphical simulation in NetAnim.   

References

[1] https://www.nsnam.org/docs/release/3.26/doxygen/ index.html [2] https://www.opennetworking.org/images/stories/ downloads/sdn-resources/onf-specications/openflow/ openflow-switch-v1.5.0.noipr.pdf [3] http://www.lrc.ic.unicamp.br/ofswitch13/ofswitch13.pdf The source repository can be downloaded from: https://bitbucket.org/yashsquare/ns3_support

#include // extra header le for Network Animator in ns3 AnimationInterface::SetConstantPosition (hosts.Get(0),50,50); AnimationInterface::SetConstantPosition (hosts.Get(1),10,60); AnimationInterface::SetConstantPosition (hosts.Get (240, 25); AnimationInterface anim (“ofs13-modify.xml”);

By: Radha Govani, Yash Modi and Jitendra Bhatia The above four lines of code will set the position of the hosts (nodes) at the given coordinates on the X-Y plane (refer to the screenshot in Figure 6), and then generate the ofs13modify.xml le for the ofswitch13-modify.cc . In Figure 6, the node with the IP address 10.100.1 (the upper left corner) represents OFSwitch with SDN controller.

34

Radha Govani and Yash Modi are open source enthusiasts. You can contact them at [email protected] and [email protected]. Jitendra Bhatia works as assistant professor at Vishwakarma Government Engineering College. You can contact him at [email protected].


Let’s Try Admin

Taming the Cloud: Provisioning with Terraform Terraform is open source software that enables sysadmins and developers to write, plan and create infrastructure as code. It is a no-frills software package, which is very simple to set up. It uses a simple configuration language or JSON, if you wish.

T

erraform is a tool to create and manage infrastructure that works with various IaaS, PaaS and SaaS service providers. It is very simple to set up and use, as there aren’t multiple packages, agents and servers, etc, involved. You just declare your infrastructure in a single (or multiple) le using a simple conguration language (or JSON), and that’s it. Terraform takes your congurations, evaluates the various building blocks from those to create a dependency graph, and presents you a plan to create the infrastructure. When you are satised with the creation plan, you apply the congurations and Terraform creates independent resources in parallel. Once some infrastructure is created using Terraform, it compares the current state of the infrastructure with the declared congurations on subsequent runs, and only acts upon the changed part of the infrastructure. Essentially, it is a CRUD (Create Read Update Destroy) tool and acts on the infrastructure in an idempotent manner.

it anywhere in your executable’s search path and all is ready to run. The following script could be used to download, unzip and verify the set-up on your GNU/Linux or Mac OS X nodes: HCTLSLOC=’/usr/local/bin’ HCTLSURL=’https://releases.hashicorp.com’ # use latest version shown on https://www.terraform.io/ downloads.html TRFRMVER=’x.y.z’ if uname -v | grep -i darwin 2>&1 > /dev/null then OS=’darwin’ else OS=’linux’ f

wget -P /tmp --tries=5 -q -L “${HCTLSURL}/

Installation and set-up

terraform/${TRFRMVER}/terraform_${TRFRMVER}_${OS}_amd64.zip”

Terraform is created in Golang, and is provided as a static binary without any install dependencies. You just pick the correct binary (for GNU/Linux, Mac OS X, Windows, FreeBSD, OpenBSD and Solaris) from its download site, unzip

sudo unzip -o “/tmp/terraform_${TRFRMVER}_${OS}_amd64.zip” -d “${HCTLSLOC}” rm -fv “/tmp/terraform_${TRFRMVER}_${OS}_amd64.zip” terraform version


35

Admin

Let’s Try

Concepts that you need to know You only need to know a few concepts to start using Terraform quickly to create the infrastructure you desire. ‘Providers’ are some of the building blocks in Terraform which abstract different cloud services and back-ends to actually CRUD various resources. Terraform gives you different providers to target different service providers and back-ends, e.g., AWS, Google Cloud, Digital Ocean, Docker and a lot of others. You need to provide different attributes applicable to the targeted service/ back-end like the access/secret keys, regions, endpoints, etc, to enable Terraform to create and manage various cloud/ back-end resources. Different providers offer various resources which correspond to different building blocks, e.g., VMs, storage, networking, managed services, etc. So only a single provider is required to make use of all the resources implemented in Terraform, to create and manage infrastructure for a service or back-end. There are ‘provisioners’ that correspond to different resources to initialise and congure those resources after their creation. The provisioners mainly do tasks like uploading les, executing remote/local commands/scripts, running conguration management clients, etc. You need to describe your infrastructure using a simple configuration language in single or multiple files, all with the .tf extension. The configuration model of Terraform is declarative, and it mainly merges all the .tf files in its working directory at runtime. It resolves the dependencies between various resources by itself to create the correct final dependency graph, to bring up independent resources in parallel. Terraform could use JSON as well for its configuration language, but that works better when Terraform configurations are generated by automated tools. The Terraform format is more human-readable and supports comments, so you could mix and match .tf and .json configuration files in case some things are human coded and others are tool generated. Terraform also provides the concepts of variables, and functions working on those variables, to store, assign and transform various things at runtime. The general workflow of Terraform consists of two stages —to plan and apply. The plan stage evaluates the merged (or overridden) configs, and presents a plan before the operator about which resources are going to get created, modified and deleted. So the changes required to create your desired infrastructure are pretty clear at the plan stage itself and there are no surprises at runtime. Once you are satisfied with the plan generated, the apply stage initiates the sequence to create the resources required to build your declared infrastructure. Terraform keeps a record of the created infra in a state file (default, terraform.tfstate ) and on every further plan-and-apply cycle, it compares the current state of the infra at runtime with the cached state. After

36


the comparison of states, it only shows or applies the difference required to bring the infrastructure to the desired state as per its configuration. In this way, it creates/maintains the whole infra in an idempotent manner at every apply stage. You could mark vari ous resources manually to get updated in the next apply phase using the taint operation. You could also clean up the infra created, partially or fully, with the destroy operation.

Working examples and usage Our rst example is to clarify the syntax for various sections in Terraform conguration les. Download the code example1.tf from http://opensourceforu.com/ article_source_code/sept17/terraform.zip . The code is a template to bring up multiple instances of AWS EC2 VMs with Ubuntu 14.04 LTS and an encrypted EBS data volume, in a specied VPC subnet, etc. The template also does remote provisioning on the instance(s) brought up by transferring a provisioning script and doing some remote execution. Now, let’s dissect this example, line by line, in order to practically explore the Terraform concepts. The lines starting with the keyword variable are starting the blocks of input variables to store values. The variable blocks allow the assigning of some initial values used as default or no values at all. In case of no default values, Terraform will prompt for the values at runtime, if these values are not set using the option -var ‘=’ . So, in our example, sensitive data like AWS access/private keys are not b eing put in the template as it is advisable to supply these at runtime, manually or through the command options or through environment variables. The environment variables should be in the form of TF_VAR_name to let Terraform read it. The variables could hold string, list and map types of values, e.g., storing a map of different amis and subnets for different AWS regions as demonstrated in our example. The string value is contained in double quotes, lists in square brackets and maps in curly braces. The variables are referenced, and their values are extracted through interpolation at different places using the syntax ${var.} . You could explore everything about Terraform variables on the official variables help page. It’s easy to guess that the block starting with the keyword provider is declaring and supplying the arguments for the service/back-end. The different providers take different arguments based upon the service/back-end being used and you could explore those in detail on the ofcial providers page. The resource keyword contains the main meat in any Terraform conguration. We are using two AWS building blocks in our example: aws_instance to bring up instances

Let’s Try Admin and aws_route53_record to create cname records for the instances created. Every resource block takes up some arguments to customise the resource(s) it creates and exposes some attributes of the resource(s) created. Each resource block starts with resource , and the important thing is that the combination should be unique in the same Terraform conguration scope. The prex of each resource is linked to its provider, e.g., all the AWS prex resources require an AWS provider. The simple form of accessing the attribute of a resource is ... Our example shows that the public_ip and public_dns attributes of the created instances are being accessed in route53 and output blocks. Some of the resources require a few post-creation actions like connecting and running local and/or remote commands, scripts, etc, on AWS instance(s). The connection block is declared to connect to that resource, e.g., by creating a ssh connection to the created instances in our example. The provisioner blocks are the mechanisms to use the connection to upload le(s) and the directory to the created resource(s). The provisioners also run local or remote commands, while Chef runs concurrently. You could explore those aspects in detail on the ofcial provisioners help page. Our example is uploading a provisioning script and kicking that off remotely over ssh to provision the created instances out-of-the-box. Terraform provides some meta-parameters available to all the resources, like the count argument in our example. The count.index keeps track of the current resource being created to reference that now or later, e.g., we are creating a unique name tag for each instance created, in our example. Terraform deducts the proper dependencies as we are referencing the attribute of aws_instance in aws_route53_record; so it creates the instances before creating their cname records. You could use meta-variable depends_on in cases where there is no implicit dependency between resources and you want to ensure that explicitly. The above-mentioned variables help the page provide detailed information about the metavariables too. The last block declared in our example conguration is the output block. As is evident by the name itself, the output could dump the raw or transformed attributes of the resources created, on demand, at any time. You can also see the usage of various functions like the format and the element in the example conguration. These functions transform the variables into other useful forms, e.g., the element function is retrieving the correct public_ip based upon the current index of the instances created. The ofcial interpolation help page provides detailed information about the various functions provided by Terraform. Now let’s look at how to decipher the output being dumped when we invoke different phases of the Terraform workow. We’ll observe the following kind of output if

we execute the command terraform plan -var ‘num_ nds=”3”’ after exporting the TF_VAR_aws_access_key and TF_VAR_aws_access_key, in the working directory where the rst example cong was created: + aws_instance.test.0 ... + aws_instance.test.1 ... + aws_instance.test.2 ... + aws_route53_record.test.0 ... + aws_route53_record.test.1 ... + aws_route53_record.test.2 Plan: 6 to add, 0 to change, 0 to destroy.

If there is some error in the conguration, then that will come up in the plan phase only and Terraform dumps the parsing errors. You can explicitly verify the conguration for any issue using the terraform validate command. If all is good, then the plan phase dumps the resources it’s going to create (indicated by the + sign before the resources’ names, in green colour) to converge to the declared model of the infrastructure. Similarly, the Terraform plan output represents the resources it’s going to delete in red (indicated by the – sign) and the resources it will update in yellow (indicated by the ~ sign). Once you are satised with the plan of resources creation, you can run terraform apply to apply the plan and actually start creating the infrastructure. Our second example is to get you more comfortable with Terraform, and use its advanced features to create and orchestrate some non-trivial scenarios. The code example2.tf can be downloaded from http:// opensourceforu.com/article_source_code/sept17/ terraform.zip. It actually automates the task of bringing up a working cluster out-of-the-box. It brings up a congurable number of multi-disk instances from the cluster payload AMI, and then initiates a specic order of remote provisioners using null_resource, some provisioners on all the nodes and some only on a specic one, respectively. In the example2.tf template, multiple null_resource are triggered in response to the various resources created, on which they depend. In this way, you can see how easily we can orchestrate some not-so-trivial scenarios. You can also see the usage of depends_on meta-variable to ensure a dependency sequence between various resources. Similarly, you can mark those resources created by Terraform that you want to destroy www.OpenSourceForU.com | OPEN SOURCE FOR YOU | SEPTEMBER 2017 |

37

Admin

Let’s Try

or those resources that you wish to create afresh using the commands terraform destroy and terraform taint, respectively. The easy way to get quick information about the Terraform commands and their options/arguments is by typing terraform and terraform -h. The recent versions of Terraform have started to provide data sources, which are the resources to gather dynamic information from the various providers. The dynamic information gathered through the data sources is used in the Terraform congurations, most commonly using interpolation. A simple example of a data source is to gather the ami id for the latest version of an ami and use that in the instance provisioning congurations as shown below:

module to use the same code to provision test and/or production clusters. The usage of the module is simply supplying the required variables to it in the manner shown below (after running terraform get to create the necessary link for the module code): module “myvms” { source

= “../modules/awsvms”

ami_id

= “${var.ami_id}”

inst_type = “${var.inst_type}” key_name = “${var.key_name}” subnet_id = “${var.subnet_id}”

data “aws_ami” “myami” {

sg_id

= “${var.sg_id}”

num_nds

= “${var.num_nds}”

hst_env

= “${var.hst_env}”

apps_pckd = “${var.apps_pckd}”

most_recent = true

hst_rle

= “${var.hst_rle}”

root_size = “${var.root_size}” flter {

swap_size = “${var.swap_size}”

name = “name”

vol_size = “${var.vol_size}”

values = [“MyBaseImage”]

zone_id

}

= “${var.zone_id}”

prov_scrpt= “${var.prov_scrpt}”

}

sub_dmn

= “${var.sub_dmn}”

} resource “aws_instance” “myvm” { ami = “${data.aws_ami.myami.id} … }

You also need to create a variables.tf in the location of your module source, requiring the same variables you ll in your module. Here is the module variables.tf to pass the variables supplied from the caller of the module:

Code organisation and reusability Although our examples show the entire declarative configuration in a single file, we should break it into more than one f ile. You could break your whole config into various separate configs based upon the respective functionality they provide. So our first example could be broken into variables.tf that keeps all the variables blocks, aws.tf that declares our provider, instances. tf that declares the layout of the AWS VMs, route53. tf that declares the aws route 53 functionality, and output.tf for our outputs. To keep things simple, use and maintain, keep everything related to a whole task being solved by Terraform in a single directory along with sub-directories that are named as files, scripts, keys, etc. Terraform doesn’t enforce any hierarchy of code organisation, but keeping each high level functionality in its dedicated directory will save you from unexpected Terraform actions in spite of unrelated configuration changes. Remember, in the s oftware world, “A little copying is better than a little dependency,” as things get fragile and complicated easily with each added functionality. Terraform provides the functionality of creating modules to reuse the configs created. The cluster creation template shown above is actually put in a

38

variable “ami_id” {} variable “inst_type” {} variable “key_name” {} variable “subnet_id” {} variable “sg_id” {} variable “num_nds” {} variable “hst_env” {} variable “apps_pckd” {} variable “hst_rle” {} variable “root_size” {} variable “swap_size” {} variable “vol_size” {} variable “zone_id” {} variable “prov_scrpt” {} variable “sub_dmn” {}

The Terraform ofcial documentation consists of a few detailed sections for modules usage and creation, which should provide you more information on everything related to modules.

Importing existing resources As we have seen earlier, Terraform caches the properties of the resources it creates into a state file, and by default doesn’t know about the resources not created through it.


Let’s Try Admin But recent versions of Terraform have introduced a feature to import existing resources not created through Terraform into its state file. Currently, the import feature only updates the state file, but the user needs to create the configuration for the imported resources. Otherwise, Terraform will show the imported resources with no configuration and mark those for destruction. Let’s make this clear by importing an AWS instance, which wasn’t brought up through Terraform, into some Terraform-created infrastructure. You need to run the command terraform import aws_instances. in the directory where a Terraform state le is located. After the successful import, Terraform gathers information about the instance and adds a corresponding section in the state le. If you see the Terraform plan now, it’ll show something like what follows: - aws_instance .
Resource Name>

So it means that now you need to create a corresponding conguration in an existing or new .tf le. In our example, the following Terraform section should be enough to not let Terraform destroy the imported resource. resource “aws_instance” “” { ami = “” instance_type = “” tags { ... } }

Please note that you only need to mention the Terraform resource attributes that are required as per the Terraform document. Now, if you see the Terraform plan, the earlier shown destruction plan goes away for the imported resource. You could use the following command to extract the attributes of the imported resource to create its conguration:

The ofcial Terraform documentation provides clear examples to import the various resources into an existing Terraform infrastructure. But if you are looking to include the existing AWS resources in the AWS infra created by Terraform in a more automated way, then take a look at the Terraforming tool link in the References section. Note: Terraform

providers are no longer distributed as part of the main Terraform distribution. Instead, they are installed automatically as part of running terraform init. The import command requires that imported resources be specied in the conguration le. Please see terraform changelog https://github.com/hashicorp/terraform/blob/ v0.10.0/CHANGELOG.md for these.

Missing bytes You should now be feeling comfortable about starting to automate the provisioning of your cloud infrastructure. To be frank, Terraform is so feature-rich now that it can’t be fully covered in a single or multiple articles and deserves a dedicated book (which has already shaped up in the form of an ebook, ‘Terraform Up & Running’). So you could further take a look at the examples provided in its ofcial Git repo. Also, the References section offers a few pointers to some excellent reads to make you more comfortable and condent with this excellent cloud provisioning tool. Creating on-demand and scalable infrastructure in the cloud is not very difcult if some very simple basic principles are adopted and implemented using some featurerich but no-fuss, easy-to-use standalone tools. Terraform is an indispensable tool for creating and managing cloud infrastructure in an idempotent way across a number of cloud providers. It could further be glued together with some other management pieces to create an immutable infrastructure workow that can tame any kind of modern cloud infrastructure. The ‘Terraform Up and Running’ ebook is already out in the form of a print book.

References sed -n ‘/aws_instance./,/}/p’ terraform.tfstate | \ grep -E ‘ami|instance_type|tags’ | grep -v ‘%’ | sed ‘s/^ *//’ | sed ‘s/:/ =/’

Please pay attention when you import a resource into your current Terraform state and decide not to use that going forward. In which case, don’t forget to rename your terraform.state.backup as terraform.state le to roll back to the previous state. You could also delete that resource block from your state le, as an alternative, but it’s not a recommended approach. Otherwise, Terraform will try to delete the imported but not desired resource and that could be catastrophic in some cases.

[1] Terraform examples: https://github.com/hashicorp/ terraform/tree/master/examples [2] Terraforming tool: https://github.com/dtan4/terraforming [3] A Comprehensive Guide to Terraform: https://blog. gruntwork.io/a-comprehensive-guide-to-terraformb3d32832baca#.ldiays7wk [4] Terraform Up & Running: http://www. terraformupandrunning.com/?ref=gruntwork-blogcomprehensive-terraform

By: Ankur Kumar The author is a systems and infrastructure developer/ architect and FOSS researcher, currently based in the US. You can nd some of his other writings on FOSS at: https://github.com/richnusgeeks.


39

Admin

Let’s Try

Visualising the Response Time of a Web Server Using Wireshark The versatile Wireshark tool can be put to several uses. This article presents a tutorial on using Wireshark to discover and visualise the response time of a Web server.

W

ireshark is a cross-platform network analysis tool used to capture packets in real-time. Wireshark includes lters, ow statistics, colour coding, and other features that allow you to get a deep insight into network trafc and to inspect individual packets. Discovering the delayed HTTP responses for a particular HTTP request from a particular PC is a tedious task for most admins. This tutorial will teach readers how to discover and visualise the response time of a Web server using Wireshark. OSFY has published many articles on Wireshark, which you can refer to for a better understanding of the topic. Step 1: Start capturing the packets using Wireshark on a specied interface to which you are connected. Refer to the

bounding box in Figure 1 for available interfaces. In this tutorial, we are going to capture Wi-Fi packets, so the option ‘Wi-Fi’ has been selected (if you wish to capture the packets using Ethernet or any other interface, select the corresponding options). Step 2: Here, we make a request to http://www. wikipedia.org and, as a result, Wikipedia sends an HTTP response of ‘200 OK’, which indicates the requested action was successful. ‘200 OK’ implies that the response contains a payload, which represents the status of the requested resource (the request is successful). Now filter all the HTTP packets as shown in Figure 2, as follows: syntax: http

Figure 1: Interface selection

40

Figure 2: Filtering HTTP


Let’s Try Admin Step 3: We now lter the requests and response sent from the local PC to Wikipedia and vice versa. Start ltering the IP of www.wikipedia.org (a simple traceroute or pathping can reveal the IP address of any Web server) and your local PC IP (a simple ipcong for Windows and ifcong for Linux can reveal your local PC IP). Syntax: ip.addr== 91.198.174.192 && ip.addr == 192.168.155.59

Figure 3: Allow sub-dissector to reassemble TCP streams

Step 4: In order to view the response of HTTP, right-click on any response packet (HTTP/1.1). Go to Protocol preference and then uncheck the subdissector to reassemble TCP streams (marked and shown in Figure 3). If the TCP preference ‘Allow sub-dissector to reassemble TCP streams’ is off, the http.time will be the time between the GET request and the rst packet of the response, the one containing ‘OK’. If ‘Allow sub-dissector to reassemble TCP streams’ is on and the HTTP reassembly preferences have been left at their defaults (on), http.time will be the time between the GET request and the last packet of the response. Procedure: Right-click on any HTTP response packet -> Protocol preference -> uncheck ‘Reassemble HTTP headers spanning multiple TCP segments’ and ‘Reassemble HTTP bodies spanning multiple TCP segments’. Step 5: Create a lter based on the response time as shown in Figure 4, and visualise the HTTP responses using an I/O graph as shown in Figure 5. 





Figure 4: Response time

Syntax: http.time >= 0.050000

Figure 5: Statistics --> I/O graph

Step 6: To calculate the delta (delay) time between request and response, use Time Reference (CTRL-T in the GUI) for easy delta time calculation. Step 7: In order to display only the HTTP response, add a lter http.time >=0.0500 in the display lter. The graph, as shown in Figure 6, depicts the result of the HTTP responses (delta time).

By: M. Kannan, Poomanam and Prem Latha M. Kannan is an associate professor and head of the department of electronics engineering, Madras Institute of Technology. His research interests include computer networks, VLSI, embedded systems and wireless security.

Figure 6: Visualisation of HTTP responses

Poomanam and Prema Latha are specialists in VLSI at the Madras Institute of Technology, Anna University. Their research interests include computer networks, VLSI and embedded design.


41

Admin

How To

DevOps Series Creating a Virtual Machine for Erlang/OTP Using Ansible

This seventh article in the DevOps series is a tutorial on how to create a test virtual machine (VM) to compile, build, and test Erlang/ OTP from its source code. You can then adapt the method to create different VMs for various Erlang releases.

E

rlang is a programming language designed by Ericsson primarily for soft real-time systems. The Open Telecom Platform (OTP) consists of libraries, applications and tools to be used with Erlang to implement services that require high availability. In this article, we will create a test virtual machine (VM) to compile, build, and test Erlang/OTP from its source code. This allows you to create VMs with different Erlang release versions for testing. The Erlang programming language was developed by Joe Armstrong, Robert Virding and Mike Williams in 1986 and released as free and open source software in 1998. It was initially designed to work with telecom switches, but is widely used today in large scale, distributed systems. Erlang is a concurrent and functional programming language, and is released under the Apache License 2.0.

The IP address of the guest CentOS 6.8 VM is added to the inventory le as shown below:

Setting it up

## Allows people in group wheel to run all commands

A CentOS 6.8 virtual machine (VM) running on KVM is used for the installation. Internet access s hould be available from the guest machine. The VM should have at least 2GB of RAM allotted to build the Erlang/OTP documentation. The Ansible version used on the host (Parabola GNU/Linux-libre x86_64) is 2.3.0.0. The ansible/ folder contains the following les:

%wheel ALL=(ALL)

erlang ansible_host=192.168.122.150 ansible_connection=ssh ansible_user=bravo ansible_password=password

An entry for the erlang host is also added to the /etc/hosts le as indicated below: 192.168.122.150 erlang

A ‘bravo’ user account is created on the test VM, and is added to the ‘wheel’ group. The /etc/sudoers le also has the following line uncommented, so that the ‘bravo’ user will be able to execute sudo commands:

ALL

We can obtain the Erlang/OTP sources from a stable tarball, or clone the Git repository. The steps involved in both these cases are discussed below.

Building from the source tarball ansible/inventory/kvm/inventory ansible/playbooks/confguration/erlang.yml

42

The Erlang/OTP stable releases are available at http://www. erlang.org/downloads. The build process is divided into many


Admin

How To

steps, and we shall go through each one of them. The version of Erlang/OTP can be passed as an argument to the playbook. Its default value is the release 19.0, and is dened in the variable section of the playbook as shown below:

- name: Download and extract Erlang source tarball unarchive: src: “http://erlang.org/download/{{ ERL_VERSION }}.tar. gz” dest: “{{ ERL_DIR }}” remote_src: yes

vars:

ERL_VERSION: “otp_src_{{ version | default(‘19.0’) }}” ERL_DIR: “{{ ansible_env.HOME }}/installs/erlang” ERL_TOP: “{{ ERL_DIR }}/{{ ERL_VERSION }}” TEST_SERVER_DIR: “{{ ERL_TOP }}/release/tests/test_server”

The ERL_DIR variable represents the directory where the tarball will be downloaded, and the ERL_TOP variable refers to the top-level directory location containing the s ource code. The path to the test directory from where the tests will be invoked is given by the TEST_SERVER_DIR variable. Erlang/OTP has mandatory and optional package dependencies. Let’s rst update the software package repository, and then install the required dependencies as indicated below: tasks:

- name: Update the software package repository become: true

yum: name: ‘*’

The ‘congure’ script is available in the sources, and it is used to generate the Makele based on the installed software. The ‘make’ command will build the binaries from the source code. - name: Build the project command: “{{ item }} chdir={{ ERL_TOP }}” with_items: - ./congure - make environment: ERL_TOP: “{{ ERL_TOP }}”

After the ‘make’ command nishes, the ‘bin’ folder in the top-level sources directory will contain the Erlang ‘erl’ interpreter. The Makele also has targets to run tests to verify the built binaries. We are remotely invoking the test execution from Ansible and hence -noshell -noinput are passed as arguments to the Erlang interpreter, as shown in the . yaml le.

update_cache: yes - name: Prepare tests - name: Install dependencies become: true

command: “{{ item }} chdir={{ ERL_TOP }}” with_items:

package: name: “{{ item }}”

- make release_tests environment:

state: latest

ERL_TOP: “{{ ERL_TOP }}”

with_items: - wget

- name: Execute tests

- make

shell: “cd {{ TEST_SERVER_DIR }} && {{ ERL_TOP }}/bin/erl

- gcc

-noshell -noinput -s ts install -s ts smoke_test batch -s

- perl

init stop”

- m4 - ncurses-devel - sed - libxslt - fop

The Erlang/OTP sources are written using the ‘C’ programming language. The GNU C Compiler (GCC) and GNU Make are used to compile the source code. The ‘libxslt’ and ‘fop’ packages are required to generate the documentation. The build directory is then created, the source tarball is downloaded and it is extracted to the directory mentioned in ERL_DIR.

You need to verify that the tests have passed successfully by checking the $ERL_TOP/release/tests/test_server/index. html page in a browser. A screenshot of the test results is shown in Figure 1. The built executables and libraries can then be installed on the system using the make install command. By default, the install directory is /usr/local. - name: Install command: “{{ item }} chdir={{ ERL_TOP }}” with_items: - make install become: true

- name: Create destination directory

environment:

le: path=”{{ ERL_DIR }}” state=directory

44



How To Admin

server” tasks: - name: Update the software package repository become: true

yum: name: ‘*’ update_cache: yes - name: Install dependencies become: true

package: name: “{{ item }}” state: latest

with_items: - wget - make - gcc - perl - m4

- ncurses-devel

Figure 1: Test results

- sed

The documentation can also be generated and installed as shown below:

- libxslt - fop

- name: Make docs


shell: “cd {{ ERL_TOP }} && make docs”


environment: ERL_TOP: “{{ ERL_TOP }}” FOP_HOME: “{{ ERL_TOP }}/fop”

- name: Download and extract Erlang source tarball

FOP_OPTS: “-Xmx2048m”

unarchive: src: “http://erlang.org/download/{{ ERL_VERSION

}}.tar.gz” - name: Install docs

dest: “{{ ERL_DIR }}”

become: true

remote_src: yes

shell: “cd {{ ERL_TOP }} && make install-docs” environment:

- name: Build the project


command: “{{ item }} chdir={{ ERL_TOP }}”

The total available RAM (2GB) is specifed in the FOP_OPTS environment variable. The complete playbook to download, compile, execute the tests, and also generate the documentation is given below:

with_items: - ./congure - make

environment: ERL_TOP: “{{ ERL_TOP }}”

--- name: Setup Erlang build

- name: Prepare tests

hosts: erlang gather_facts: true


tags: [release]

- make release_tests

vars:

with_items: environment: ERL_TOP: “{{ ERL_TOP }}”

ERL_VERSION: “otp_src_{{ version | default(‘19.0’) }}” ERL_DIR: “{{ ansible_env.HOME }}/installs/erlang” ERL_TOP: “{{ ERL_DIR }}/{{ ERL_VERSION }}” TEST_SERVER_DIR: “{{ ERL_TOP }}/release/tests/test_

- name: Execute tests shell: “cd {{ TEST_SERVER_DIR }} && {{ ERL_TOP }}/bin/ erl -noshell -noinput -s ts install -s ts smoke_test batch -s


45

Admin

How To

init stop”

become: true

- name: Install

name: “{{ item }}”


with_items:

state: latest

- make install

package:

with_items: - wget

become: true

- make

environment:

- gcc


- perl - m4

- name: Make docs

- ncurses-devel

shell: “cd {{ ERL_TOP }} && make docs”

- sed

environment:

- libxslt


- fop

FOP_HOME: “{{ ERL_TOP }}/fop”

- git

FOP_OPTS: “-Xmx2048m”

- autoconf

- name: Install docs


become: true


shell: “cd {{ ERL_TOP }} && make install-docs”


- name: Clone the repository

git: repo: “https://github.com/erlang/otp.git”

The playbook can be invoked as follows:

dest: “{{ ERL_DIR }}/otp”

$ ansible-playbook -i inventory/kvm/inventory playbooks/

- name: Build the project

conguration/erlang.yml -e “version=19.0” --tags “release” -K


with_items:

Building from the Git repository

- ./otp_build autoconf

We can build the Erlang/OTP sources from the Git repository too. The complete playbook is given below for reference:

- ./congure - make


- name: Setup Erlang Git build hosts: erlang gather_facts: true tags: [git] vars: GIT_VERSION: “otp”

The ‘git’ and ‘autoconf’ software packages are required for downloading and building the sources from the Git repository. The Ansible Git module is used to clone the remote repository. The source directory provides an otp_build script to create the confgure script. You can invoke the above playbook as follows:

ERL_DIR: “{{ ansible_env.HOME }}/installs/erlang” ERL_TOP: “{{ ERL_DIR }}/{{ GIT_VERSION }}”

$ ansible-playbook -i inventory/kvm/inventory playbooks/

TEST_SERVER_DIR: “{{ ERL_TOP }}/release/tests/test_

conguration/erlang.yml --tags “git” -K

server” tasks: - name: Update the software package repository

You are encouraged to read the complete installation documentation at https://github.com/erlang/otp/blob/master/ HOWTO/INSTALL.md.

become: true

yum: name: ‘*’ update_cache: yes

By: Shakthi Kannan The author is a free software enthusiast and blogs at shakthimaan.com.

- name: Install dependencies

46


Overview

Admin

An Introduction to govcsim (a vCenter Server Simulator) govcsim is a vCenter Server and ESXi API based simulator that offers a quick fix solution for prototyping and testing code. It simulates the vCenter Server model and can be used to create data centres, hosts, clusters, etc.

ovcsim (a vCenter Server simulator) is an open source vCenter Server and ESXi API based simulator written in the Go language, using the govmomi the govmomi library. library. govcsim simulates the vCenter Server model by creating various vCenter related objects like data centres, hosts, clusters, resource pools, networks and datastores. If you are a software developer or quality engineer who works with vCenter and related technologies, then you can use govcsim for fast prototyping and for testing your code. In this article, we will write an Ansible Playbook to gather all VMs installed on a given govcsim installation. Ansible provides many modules for managing and maintaining VMware resources. (You (You can nd out more about Ansible modules for managing VMware at http://docs.ansible.com/ansible/ list_of_cloud_modules.html#vmware.) Do list_of_cloud_modules.html#vmware.) Do note that govcsim will simulate almost the identical environments provided by VMWare vCenter and ESXi server.

g

The requirements for installing govcsim are: 1. Golang 1.7+ 2. Git Step 1: Installing Golang To install the Go tools, type the following command at the terminal: $ sudo dnf install -y golang

Step 2: Confguring the Golang workspace Use the following commands to congure the Golang workspace: $ mkdir -p $HOME/go $ echo ‘export GOPATH=$HOME/go’ GOPATH=$HOME/go’ >> $HOME/.bash $HOME/.bashrc rc $ source $HOME/.bash $HOME/.bashrc rc

Installation

Check if everything is working by using the command given below:

We will use Fedora 26 for the installation of govcsim. Let’s assume that Ansible has been already installed using dnf or a source tree.

$ go env GOPATH


47

S K C A R T

Open Source and You (Success Stories)

Application Development Day (talks on hybrid app development for th for the e en ente terp rpri rise se us use) e)

Cyber Security Day

Database Day

Open Source in IoT

Cloud and Big Data

Container Day

FREEZE YOUR CALENDER NO NOW! W!

www.opensourceindia www.opensourceindia.in .in

Asia’s #1 Conf Asia’s Confer erence ence on Open Source

Regist egister er No Now! w! http://register.opensourceindia.in

Stall Booking & Partnering Opportunit Oppor tunities ies Open Now For more details, details, call Omar on +91 +91 995 8855 8855 993 or write to info@osida info@osidays. ys.com. com.

Gold Partner

Silver Partner

Associate Partners

Admin Overview

Figure 3: Ansible Playbook to get details about the virtual machine

Figure 1: Getting help from vcsim

Figure 2: Starting vcsim without any parameters

This should return your home directory path with the Go workspace. Step 3: Download govcsim using the ‘go get’ command Figure 4: Ansible in action $ go get github.com/vmware/govmomi/vcsim

Ansible at https://docs.ansible.com/ansible/. After running the playbook from Figure 3, you will get a list of virtual machine objects that are simulated by the govcsim server (see Figure 4). You can play around and write different playbooks to get information about govcsim simulated VMware objects.

$ $GOPATH/bin/vcsim -h

If everything is congured correctly, you will be able to get the help options related to govcsim. To start govcsim without any argument, use the following command: $ vcsim

References [1] Ansible documentation: https://docs.ansible.com/ansible [2] Govcsim: https://github.com/vmware/govmomi/tree/ master/vcsim

Now, govcsim is working. You can check out the various methods available by visiting https://127.0.0.1:8989/about on your favourite browser.

By: Abhijeet Kasurde

Testing govcsim with Ansible

The author works at Red Hat and is a FOSS evangelist. He loves to explore new technologies and software. You can contact him at [email protected].

Now, let’s try to write a simple Ansible Playbook, which will list down all VMs emulated by govcsim. The complete code is given in Figure 3. You can read up more about

THE COMPLETE MAGAZINE ON OPEN SOURCE

The latest from the Open Source world is here.

OpenSourceForU.com Join the community at facebook.com/opensourceforu Follow us on Twitter @OpenSourceForU

50


Insight Admin

Serverless Architectures:

Demystifying Serverless Computing Serverless architectures refer to applications that depend a lot on third party services known as BaaS (Backend as a Service), or on custom code which runs on FaaS (Function as a Service).

I

n the 1990s, Neal Ford (now at ThoughtWorks) was working in a small company that focused on a technology called Clipper. By writing an object-oriented framework based on Clipper, DOS applications were built using dBase. With the expertise the rm had on Clipper, it ran a thriving training and consulting business. Then, all of a sudden, this Clipper-based business disappeared with the rise of Windows. So Neal Ford and his team went scrambling to learn and adopt new technologies. “Ignore the march of technology at your peril,” is the lesson that one can learn from this experience. Many of us live inside ‘technology bubbles’. It is easy to get cozy and lose track of what is happening around us. All of a sudden, when the bubble bursts, we are left s crambling to nd a new job or business. Hence, it is important to stay relevant. In the 90s, that meant catching up with things like graphical user interfaces (GUIs), client/server technologies and later, the World Wide Web. Today, relevance is all about being agile and leveraging the cloud, machine learning, articial intelligence, etc. With this background, let’s delve into serverless computing, which is an emerging eld. In this article, readers will learn how to employ the serverless approach in their applications and discover key serverless technologies; we

will end the discussion by looking at the limitations of the serverless approach.

Why serverless? Most of us remember using server machines of one form or another. We remember logging remotely to server machines and working with them for hours. We had cute names for the servers - Bailey, Daisy, Charlie, Ginger, and Teddy - treating them well and taking care of them fondly. However, there were many problems in using physical servers like these: Companies had to do capacity planning and predict their future resource requirements. Purchasing servers meant high capital expenses (capex) for companies. We had to follow lengthy procurement processes to purchase new servers. We had to patch and maintain the servers … and so on. The cloud and virtualisation provided a level of exibility that we hadn’t known with physical servers. We didn’t have to follow lengthy procurement processes, or worry about who ‘owns the server’, or why only a particular team had ‘exclusive access to that powerful server’, etc. The task of procuring physical machines became obsolete with the arrival 








51

Admin

Insight

of virtual machines (VMs) and the cloud. The architecture The solution is to set up some compute capacity to process we used also changed. For example, instead of scaling up data from a database and also execute this logic in a language by adding more CPUs or memory to physical servers, we of choice. For example, if you are using the AWS platform, started ‘scaling out’ by adding more machines as needed, you can use DynamoDB for the back-end, write programming but, in the cloud. This model gave logic as Lambda functions, and us the exibility of an opex-based expose them through the AWS API AWS Lambda (operational expenses-based) Gateway with a load balancer. This revenue model. If any of the VMs entire set-up does not require you to Key went down, we got new VMs provision any infrastructure or have Apache Serverless MS Azure spawned in minutes. In short, we any knowledge about underlying OpenWhisk Functions Platforms started treating servers as ‘cattle’ servers/VMs in the cloud. You can and not ‘pets’. use a database of your choice for However, the cloud and the back-end. Then choose any Google Cloud virtualisation came with their own programming language supported Functions problems and still have many in AWS Lambda, including Java, limitations. We are still spending a Python, JavaScript, and C#. There is Figure 1: Key serverless platforms lot of time managing them — for no cost involved if there aren’t any example, bringing VMs up and down, based on need. We have users using the MovieBot. If a blockbuster like ‘Baahubali’ is to architect for availability and fault-tolerance, size workloads, released, then there could be a huge surge in users accessing and manage capacity and utilisation. If we have dedicated VMs the MovieBot at the same time, and the set-up would provisioned in the cloud, we still have to pay for the reserved effortlessly scale (you have to pay for the calls, though). resources (even if it’s just idle time). Hence, moving from a Phew! You essentially engineered a serverless application. capex model to an opex one is not enough. What we need is to With this, it’s time to dene the term ‘serverless’. only pay for what we are using (and not more than that) and Serverless architectures refer to applications that signicantly ‘pay as you go’. Serverless computing promises to address depend on third-party services (known as Backend-as-aexactly this problem. Service or BaaS) or on custom code that’s run in ephemeral The other key aspect is agility. Businesses today need containers (Function-as-a-Service or FaaS). to be very agile. Technology complexity and infrastructure Hmm, that’s a mouthful of words; s o let’s dissect operations cannot be used as an excuse for not delivering this description. value at scale. Ideally, much of the engineering effort should Backend-as-a-Service: Typically, databases (often NoSQL be focused on providing functionality that delivers the avours) hold the data and can be accessed over the cloud, desired experience, and not in monitoring and managing the and a service can be used to help access that back-end. infrastructure that supports the scale requirements. This is Such a back-end service is referred to as BaaS. where serverless shines. Function-as-a-Service: Code that processes the requests (i.e., the ‘programming logic’ written in your favourite What is serverless? programming language) could be run on containers that are Consider a chatbot for booking movie tickets - let’s call it spun and destroyed as needed. They are known as FaaS. MovieBot. Any user can make queries about movies, book The word ‘serverless’ is misleading because it literally tickets, or cancel them in a conversational style (e.g., “Is means there are no servers. Actually, the word implies, “I don’t ‘Dunkirk’ playing in Urvashi Theatre in Bengaluru tonight?” care what a server is.” In other words, serverless enables us in voice or text). to create applications without thinking about servers, i.e., we This solution requires three elements: a chat interface can build and run applications or services without worrying channel (like Skype or Facebook Messenger), a natural about provisioning, managing or scaling the underlying language processor (NLP) to understand the user’s intentions infrastructure. Just put your code in the cloud and run it! Keep (e.g., ‘book a ticket’, ‘ticket availability’, ‘cancellation’, etc), in mind that this applies to Platform-as-a-Service (PaaS) as and then access to a back-end where the transactions and data well; although you may not deal with direct VMs with PaaS, pertaining to movies is stored. The chat interface channels you still have to deal with instance sizes and capacity. are universal and can be used for different kinds of bots. NLP Think of serverless as a piece of functionality to run can be implemented using technologies like AWS Lex or IBM — not in your machine but executed remotely. Typically, Watson. The question is: how is the back-end served? Would serverless functions are executed in an ‘event-driven’ you set up a dedicated server (or a cluster of servers), an API fashion — the functions get executed as a response to events gateway, deploy load balancers, or put in place identity and or requests on HTTP. In the case of the MovieBot, the access control mechanisms? That’s costly and painful, right! Lambda functions are invoked to serve user queries as and That’s where serverless technology can help. when user(s) interact with it. 



52


Insight Admin

Use cases With serverless architecture, developers can deploy certain types of solutions at scale with cost-effectiveness. We have already discussed developing chatbots - it is a classic use case for serverless computing. Other key use cases for the serverless approach are given below. 1) Three-tier Web applications: Conventional single page applications (SPA), which rely on REpresentative State Transfer (REST) based services to perform a given functionality, can be re-written to leverage serverless functions front-ended by an API gateway. This is a powerful pattern that helps your application scale innitely, without concerns of conguring scale-out or infrastructure resources.. 2) Scalable batch jobs: Batch jobs were traditionally run as daemons or background processes on dedicated VMs. More often than not, this approach hit scalability and had reliability issues - developers would leave their critical processes with Single Points of Failure (SPoF). With the serverless approach, batch jobs can now be redesigned as a chain of mappers and reducers, each running as independent functions. Such mappers and reducers will share a common data store, something like a blob storage or a queue, and can individually scale up to meet the data processing needs. 3) Stream processing: Related to scalable batch jobs is the pattern of ingesting and processing large streams of data for near-real-time processing. Streams from services like Kafka and Kinesis can be processed by serverless functions, which can be scaled seamlessly to reduce latency and increase the throughput of the system. This pattern can elegantly handle spiky loads as well. 4) Automation/event-driven processing: Perhaps the rst application of serverless computing was automation. Functions could be written to respond to certain alerts or events. These could also be periodically scheduled to augment the capabilities for the cloud service provider through extensibility. The kind of applications that are best suited for s erverless architectures include mobile back-ends, data processing systems (real-time and batch) and Web applications. In general, serverless architecture is suitable for any distributed system that reacts to events or process workloads dynamically, based on demand. For example, serverless computing is suitable for processing events from IoT (Internet of Things) devices, processing large data sets (in Big Data) and intelligent systems that respond to queries (chatbots).

Functions has good support for a wider variety of languages and integrates with Microsoft’s Azure services. Google’s Cloud Functions is currently in beta. One of the key open source players in serverless technologies is Apache OpenWhisk, backed by IBM and Adobe. It is often tedious to develop applications directly on these platforms (AWS, Azure, Google and OpenWhisk). The serverless framework is a popular solution that aims to ease application development on these platforms. Many solutions (especially open source) focus on abstracting away the details of container technologies like Docker and Kubernetes. Hyper.sh provides a container hosting service in which you can use Docker images directly in serverless style. Kubeless from Bitnami, Fission from Platform9, and funktion from Fabric8 are serverless frameworks that provide an abstraction over Kubernetes. Given that serverless architecture is an emerging approach, technologies are still evolving and are yet to mature. So you will see a lot of action in this space in the years to come. Join us at the India Serverless Summit 2017

These are the best of times, and these are the worst of times! There are so many awesome new technologies to catch up on. But, we simply can’t. We have seen a progression of computing models - from virtualisation, IaaS, PaaS, containers, and now, serverless - all in a matter of a few years. You certainly don’t want to be left behind. So join us at the Serverless Summit, India’s first confluence on serverless technologies, being held on October 27, 2017 at Bengaluru. It is the best place to hear from industry experts, network with technology enthusiasts, as well as learn about how to adopt serverless architecture. The keynote speaker is John W illis, director of ecosystem development at Docker and a DevOps guru (widely known for the book ‘The DevOps Handbook’ that he co-authored). Open Source For You is the media partner and the Cloud Native Computing Foundation is the community partner for this summit. For more details, please visi t the website www.inserverless.com.

Challenges in going serverless Despite the fact that a few large businesses are already powered entirely by serverless technologies, we should keep in mind that serverless is an emerging approach. There are many challenges we need to deal with when developing serverless solutions. Let us discuss them in the context of the MovieBot example mentioned earlier. Debugging Unlike in typical application development, there is no concept of a local environment for serverless functions. Even fundamental debugging operations like stepping-through, breakpoints, step-over and watch points are not available with serverless functions. As of now, we need to rely on extensive logging and instrumentation for debugging. 

Serverless technologies There are many proprietary and a few open source serverless technologies and platforms available for us to choose from. AWS Lambda is the earliest (announced in late 2014 and released in 2015) and the most popular serverless technology, while other players are fast catching up. Microsoft’s Azure


53

Admin

Insight

When MovieBot provides an inconsistent response or does not understand the intent of the user, how do we debug the code that is running remotely? For situations such as this, we have to log numerous details: NLP scores, the dialogue responses, query results of the movie ticket database, etc. Then we have to manually analyse and do detective work to nd out what could have gone wrong. And, that is painful. State management Although serverless is inherently stateless, real-world applications invariably have to deal with state. Orchestrating a set of serverless functions becomes a signicant challenge when there is a common context that has to be passed between them. Any chatbot conversation represents a dialogue. It is important for the program to understand the entire conversation. For example, for the query, “Is ‘Dunkirk’ playing in Urvashi Theatre in Bengaluru tonight?” if the answer from MovieBot is “Yes”, then the next query from the user could be, “Are two tickets available?” If MovieBot conrms this, the user could say, “Okay, book it.” For this transaction to work, MovieBot should remember the entire dialogue, which includes the name of the movie, the theatre’s location, the city, and the number of tickets to book. This entire dialogue represents a sequence of s tateless function calls. However, we need to persist this state for the nal transaction to be successful. This maintenance of state external to functions is a tedious task. Vendor lock-in Although we talk about isolated functions that are executed independently, we are in practice tied to the SDK (software development kit) and the services provided by the serverless technology platform. This could result in vendor lock-in because it is difficult to migrate to other equivalent platforms. Let’s assume that we implement the MovieBot on the AWS Lambda platform using Python. Though the core logic of the bot is written as Lambda functions, we need to use other related services from the AWS platform for the chatbot to work, such as AWS Lex (for NLP), AWS API gateway, DynamoDB (for data persistence), etc. Further, the bot code may need to make use of the AWS SDK to consume the services (such as S3 or DynamoDB), and that is written using boto3. In other words, for the bot to be a reality, it needs to consume many more services from the AWS platform than just the Lambda function code written in plain Python. This results in vendor lock-in because it is harder to migrate the bot to other platforms. Other challenges Each serverless function code will typically have third party library dependencies. When deploying the serverless function, we need to deploy the third party dependency packages as well, and that increases the deployment package size. Because containers are used underneath to execute the serverless functions, the increased deployment size increases the latency to start up and execute the serverless functions. 

Further, maintaining all the dependent packages, versioning them, etc, is a practical challenge as well. Another challenge is the lack of support for widely used languages from serverless platforms. For instance, as of May 2017, you can write functions in C#, Node.js (4.3 and 6.10), Python (2.7 and 3.6) and Java 8 on AWS Lambda. How about other languages like Go, PHP, Ruby, Groovy, Rust or any others of your choice? Though there are solutions to write serverless functions in these languages and execute them, it is harder to do so. Since serverless technologies are maturing with support for a wider number of languages, this challenge will gradually disappear with time. Serverless is all about creating solutions without thinking or worrying about servers; think of it as just putting your code in the cloud and running it! Serverless is a game-changer because it shifts the way you look at how applications are composed, written, deployed and scaled. If you want signicant agility in creating highly scalable applications while remaining cost-effective, serverless is what you need. Businesses across the world are already providing highly compelling solutions using serverless computing technologies. The applications serverless has range from chatbots to real-time stream processing from IoT (Internet of Things) devices. So it is not a question of if, but rather, when you will adopt the serverless approach for your business. References



[1] ‘Build Your Own Technology Radar’, Neal Ford, http:// nealford.com/memeagora/2013/05/28/build_your_own_ technology_radar.html [2] ‘Serverless Architectures’, Martin Fowler, https:// martinfowler.com/articles/serverless.html [3] ‘Why the Fuss About Serverless?’ Simon Wardley, http://blog. gardeviance.org/2016/11/why-fuss-about-serverless.html [4] ‘Serverless Architectural Patterns and Best Practices’, Amazon Web Services, https://www.youtube.com/ watch?v=b7UMoc1iUYw Serverless technologies AWS Lambda: https://aws.amazon.com/lambda/ Azure Functions: https://functions.azure.com/ Google Cloud Functions: https://cloud.google.com/ functions/ Apache OpenWhisk: https://github.com/openwhisk Serverless framework: https://github.com/serverless/ serverless Fission: https://github.com/fssion/fssion Hyper.sh: https://github.com/hyperhq/ Funktion: https://funktion.fabric8.io/ Kubeless: http://kubeless.io/ •

•

•

•

•

•

•

•

•



56

By: Ganesh Samarthyam, Manoj Ganapathi and Srushit Repakula The authors work at CodeOps Technologies, which is a software technology, consulting and training company based in Bengaluru. CodeOps is the organiser of the upcoming India Serverless Summit, scheduled on October 27, 2017. Please check www.codeops.tech for more details.


Insight Admin

A Glimpse of Microservices with Kubernetes and Docker The microservices architecture is a variant of service oriented architecture. It develops a single application as a suite of small services, each running in its own process and communicating with lightweight mechanisms, often an HTTP resource API.

‘M

icroservices’ is a compound word made of ‘micro’ and ‘services’. As the name suggests, microservices are the small modules that provide some functionality to a system. These modules can be anything that is designed to serve some specic function. These services can be independent or interrelated with each other, based on some contract. The main function of microservices is to provide isolation between services — a separation of services from servers and the ability to run them independently, with the interaction between them based on a specic requirement. To achieve this isolation, we use containerisation, which will be discussed later. The idea behind choosing microservices is to avoid correlated failure in a system where there is a dependency between services. When running all microservices inside the same process, all services will be killed if the process is restarted. By running each service in its own process, only one service is killed if that process is restarted, but restarting the server will kill all services. By running each service on its own server, it’s easier to maintain these isolated services, though there is a cost associated with this option.

How microservices are defined The microservices architecture develops a single application as a suite of small services, each running in its own process and communicating with lightweight mechanisms, often an HTTP resource API.

Microservices is a variant of the service-oriented architecture (SOA) ar chitectural style. A SOA is a style of software design in which services are provided to the other components by application components, through a communication protocol over a network that structures an application as a collection of loosely coupled services. In the microservices architecture, services should be fine-grained and the protocols should be lightweight. The benefit of breaking down an application into different smaller services is that it improves modularity and makes the application easier to understand, develop and test. It also parallelises development by enabling small autonomous teams to develop, deploy and scale their respective services independently. These services are independently deployable and scalable. Each service also provides a kind of contract allowing for different services to be written in different programming languages. They can also be managed by different teams.

The architecture of microservices Microservices follows the service-oriented architecture in which the services are independent of users, products and technologies. This architecture allows one to build applications as suites of services that can be used by other services. This architecture is in contrast to the monolithic architecture, where the services are built as a single unit comprising a client-side user interface, databases and server-side applications in a single frame — all dependent on one another. The failure of one can bring down the whole system. The microservices architecture mainly consists of the client-side user interface, databases and server-side applications as different services that are related in some way to each other but are not dependent on each other. Each layer is independent of the other, which in turn leads to easy maintenance. The architecture is represented in Figure 2. This architecture is a form or system that is built by plugging together components, somewhat like in a real world composition where a component is a unit of software that is independently replaceable and upgradeable. These microservices are easily deployable and integrated into one another. This gives rise to the possibility of continuous integration and continuous deployment.


57

Admin

Insight

doesn’t have to install and configure AMQP HTTP User Interface complex databases nor worry about Relational DB HTTP switching between HTTP Services incompatible Module 1 Module 2 Module 3 Module 4 language toolchain HTTP HTTP versions. When an Key / Value Store app is dockerised, HTTP HTTP that complexity Database 2 Database 1 is pushed into Microservices Database Client Side User Interface Database 3 microservices - application databases containers that are easily built, Figure 1: Microservices - application databases Figure 2: Microservices architecture shared and run. It is a tool that is designed to benefit both developers and APP APP APP Containers are isolated systems administrators. A A* B Bins/ Libs

Bins/ Libs

but share the OS and, where appropriate, the bins/libraries

Bins/ Libs

....result is significantly faster deployment, much less overhead, easier migration, faster restart

VM Guest OS

Guest OS

Guest OS

Container

A P P A

A P P A *

A P P B

A P P B *

A P P B *

A P P B *

Hyprevisor (Type 2) Host OS Server

D o c k e r

Host OS Server

Figure 3: Virtual machines vs containers

What’s so good about microservices? With the advances in software architecture, microservices have emerged as a different platform compared to other software architecture. Microservices are easily scalable and are not limited to a language; so you are free to choose any language for the services. The services are loosely coupled, which in turn results in ease of maintenance and exibility, as well as reduced time in debugging and deployment.

Microservices with Docker and Kubernetes Docker is a software technology that provides containers, which are a computer virtualisation method in which the kernel of an operating system allows the existence of multiple isolated user-space instances, instead of just one. Everything required to make a piece of software run is packaged into isolated containers. With microservices, containers play the same role of providing virtual environments to different processes that are running, being deployed and undergoing testing, independently. Docker is a bit like a virtual machine, but rather than creating a whole virtual operating system, Docker allows applications to use the same kernel as the system that it’s running on and only requires applications to be shipped with things not already running on the host computer. The main idea behind using Docker is to eliminate the ‘works on my machine’ type of problems that occur when collaborating on code with co-workers. Docker

58

How well does Kubernetes go with Docker? Before starting the discussion on Kubernetes, we must rst understand orchestration, which is to arrange various components so that they achieve a desired result. It also means the process of integrating two or more applications and/or services together to automate a process, or synchronise data in real-time. The intermediate path connecting two or more services is done by orchestration, which refers to the automated arrangement, coordination and management of software containers. So what does Kubernetes do then? Kubernetes is an open source platform for automating deployments, scaling and operations of application containers across clusters of hosts, providing container-centric infrastructure. Orchestration is an idea whereas Kubernetes implements that idea. It is a tool for orchestration. It deploys containers inside a cluster. It is a helper tool that can be used to manage a cluster of containers and treat all servers as a single unit. These containers are provided by Docker. The best example of Kubernetes is the Pokémon Go App, which runs on a virtual environment of Google Cloud, in a separate container for each user. Kubernetes uses a different set-up for each OS. So if you want a tool that will overcome Docker’s limitations, you should go with Kubernetes. To conclude, we may say that microservices is growing very fast, the reason being its features of independence and isolation which give it the power to easily run, test and be deployed. This is just a small summary of microservices, about which there is a lot more to learn.


By: Astha Srivastava The author is a software developer. Her areas of expertise are C, C++, C#, Java, JavaScript, HTML and ASP.NET. She has recently started working on the basics of articial intelligence. She can be reached at [email protected].

Let’s Try Developers

Selenium: A Cost-Effective Test Automation Tool for Web Applications Selenium is a software testing framework. Test authors can write tests in it without learning a test scripting language. It automates Web based applications efficiently and provides a recording/ playback system for authoring tests.

S

elenium is a portable software-testing framework for Web applications that can operate across different browsers and operating systems. It is quite similar to HP Quick Test Pro (or QTP, now called UFT) except that Selenium focuses on automating Web based applications. Testing done using this tool is usually referred to as Selenium testing. Selenium is not just a single tool but a set of tools that helps the tester to automate Web based applications more efciently. It has four components: 1. The Selenium integrated development environment (IDE) 2. The Selenium remote control (RC) 3. WebDriver 4. The Selenium grid Selenium RC and WebDriver are merged into a single framework to form Selenium 2. Selenium 1 is also referred to as Selenium RC. Jason Huggins created Selenium in 2004. Initially, he named it JavaScriptTestRunner, and later changed this to Selenium. It is licensed under Apache License 2.0. In the following sections, we will learn about how Selenium and its components operate.

The Selenium IDE The Selenium IDE is the simplest framework in the Selenium

suite and is the easiest one to learn. It is a Firefox plugin that you can install as easily as any other plugin. It allows testers to record their actions as they go through the workow that they need to test. But it can only be used with the Firefox browser, as other browsers are not supported. The recorded scripts can be converted into various programming languages supported by Selenium, and the scripts can be executed on other browsers as well. However, for the sake of simplicity, the Selenium IDE should only be used as a prototyping tool. If you want to create more advanced test cases, either use Selenium RC or WebDriver.

Selenium RC Selenium RC or Selenium Remote Control (also known as Selenium 1.0) was the agship testing framework of the whole Selenium project for a long time. It works in a way that the client libraries can communicate with the Selenium RC server that passes each Selenium command for execution. Then the server passes the Selenium command to the browser using Selenium-Core JavaScript commands. This was the rst automated Web testing tool that allowed people to use a programming language they preferred. Selenium RC components include: 1. The Selenium server, which launches and kills the


59

Developers

Let’s Try

browser, interprets and runs the Selenese commands passed from the test program, and acts as an HTTP proxy, intercepting and verifying HTTP messages passed between the browser and Application Under Test (AUT). 2. Client libraries that provide the interface between each programming language and the Selenium RC server. Selenium RC is great for testing complex AJAX based Web user interfaces under a continuous integration system. It is also an ideal solution for users of Selenium IDE who want to write tests in a more expressive programming language than the Selenese HTML table format.

The WebDriver uses a different underlying framework, while Selenium RC uses a JavaScript Selenium-Core embedded within the browser, which has its limitations. WebDriver directly interacts with the browser without any intermediary. Selenium RC depends on a server.

Architecture The architecture of WebDriver is explained in Figure 1.

Web Application

Selenese commands Selenese is the set of Selenium commands which is used to test Web applications. The tester can test the broken links, the existence of some object on the UI, AJAX functionality, the alert window, list options and a lot more using Selenese. There are three types of commands: 1. Actions: These are commands that manipulate the state of the application. Upon execution, if an action fails, the execution of the current test is stopped. Some examples are: click(): Clicks on a link, button, checkbox or radio button. contextMenuAt (locator, coordString): Simulates the user by clicking the ‘Close’ button in the title bar of a popup window or tab. 2. Accessors: These evaluate the state of the application and store the results in variables which are used in assertions. Some examples are: assertErrorOnNext: Pings Selenium to expect an error on the next command execution with an expected message. storeAllButtons: Returns the IDs of all buttons on the page. 3. Assertions: These enable us to verify the state of an application and compare it against the expected. It is used in three modes, i.e., assert, verify and waitfor. Some examples are: waitForErrorOnNext(message): Wait for error, used with the accessor assertErrorOnNext. verifySelected (selectLocator, opti onLocator):Veries that the selected item of a drop-down satises optionSpecifer.

Selenium WebDriver Selenium WebDriver is a tool that automates the testing of Web applications and is popularly known as Selenium 2.0. It is a Web automation framework that allows you to execute your tests against different browsers. WebDriver also enables you to use a programming language in creating your test scripts. The following programming languages are supported by Selenium WebDriver: 1. Java 2. .NET 3. PHP 4. Python 5. Perl 6. Ruby

Selenium Web Driver

Selenium Test (Java, C#, Ruby, Python, Perl, Php, Java Script) Figure 1: Architecture of Selenium WebDriver

The differences between WebDriver and Selenium RC are given in Table 1. Table 1 WebDriver

Selenium RC

Architecture is simpler, as it controls the browser from the OS level. It supports HtmlUnit. WebDriver is faster, as it interacts directly with the browser.

Architecture is complex, as it depends on the server.

Less object-oriented APIs and cannot be used for mobile testing.

Purely object-oriented and can be used for iPhone/Android application testing.

WebDriver is not ready to support new browsers and does not have a built-in command for the automatic generation of test results.

Selenium RC can support new browsers and have built-in commands.

It does not support HtmlUnit. It is slower, as it uses JavaScript to interact with RC.

Continued to page 64.... 60



Using the Spring Boot Admin UI for Spring Boot Applications

Using Spring Boot makes it easy for developers to create standalone, production-grade Spring based applications that can be ‘just run’. Spring Boot is a part of microservices development.

A

s part of developing microservices, many of us use the features of Spring Boot along with Spring Cloud. In the microservices world, we may have many Spring Boot applications running on the same or different hosts. If we add SpringActuator (http://docs.spring.io/ spring-boot/docs/current/reference/htmlsingle/#productionready) to the Spring Boot applications, we get a lot of out-ofthe-box end points to monitor and interact with applications. The list is given in Table 1. The end points given in Table 1 provide a lot of insights about the Spring Boot application. But if you have many applications running, then monitoring each application by hitting the end points and inspecting the JSON response is a tedious process. To avoid this hassle, the Code Centric team came up with the Spring Boot Admin (https://github. com/codecentric/spring-bootFigure 1: Spring Boot logo admin) module, which provides

us an Admin UI dashboard to administer Spring Boot applications. This module crunches the data from Actuator end points, and provides insights about all the registered applications in a single dashboard. We will demonstrate the Spring Boot admin features in the following sections. As a frst step, create a Spring Boot application that will be a Spring Boot Admin Server module by adding the Maven dependencies given below:

de.codecentric

spring-boot-admin-server

1.5.1

de.codecentric

spring-boot-admin-server-ui

1.5.1


61

Developers

Let’s Try

Table 1

ID

Description

Sensitive default

actuator

Provides a hypermedia-based ‘discovery page’ for the other endpoints. Requires Spring HATEOAS to be on the classpath. Exposes audit events information for the current application. Displays an auto-configuration report showing all auto-configuration candidates and the reason why they ‘were’ or ‘were not’ applied.

True

beans configprops dump

Displays a complete list of all the Spring Beans in your application. Displays a collated list of all @ConfigurationProperties. Performs a thread dump.

True True True

env flyway health

Exposes properties from Spring’s ConfigurableEnvironment. Shows any Flyway database migrations that have been applied. Shows application health information (when the application is secure, a simple ‘status’ when accessed over an unauthenticated connection or full message details when authenticated). Displays arbitrary application information. Shows and modifies the configuration of loggers in the application. Shows any Liquibase database migrations that have been applied. Shows ‘metrics’ information for the current application. Displays a collated list of all @RequestMapping paths. Allows the application to be gracefully shut down (not enabled by default). Displays trace information (by default, the last 100 HTTP requests).

True True False

auditevents autoconfig

info loggers liquibase metrics mappings shutdown trace

Add the Spring Boot Admin Server conguration by adding @EnableAdminServer to your conguration, as follows:

True True

False True True True True True True

throws Exception { // Page with login form is served as /login.html and does a POST on /login

package org.samrttechie;

http.formLogin().loginPage(“/login.

html”).loginProcessingUrl(“/login”).permitAll(); import org.springframework.boot.SpringApplication; import org.springframework.boot.autocongure.

// The UI does a POST on /logout on logout

http.logout().logoutUrl(“/logout”);

SpringBootApplication; import org.springframework.context.annotation.Conguration;

// The ui currently doesn’t support csrf

http.csrf().disable();

import org.springframework.security.cong.annotation.web. builders.HttpSecurity;

// Requests for the login page and the static

import org.springframework.security.cong.annotation.web.

assets are allowed

conguration.WebSecurityCongurerAdapter;

http.authorizeRequests() .antMatchers(“/login.html”, “/**/*.

import de.codecentric.boot.admin.cong.EnableAdminServer;

css”, “/img/**”, “/third-party/**”)

.permitAll();

@EnableAdminServer

// ... and any other request needs to be authorized

@Conguration

http.authorizeRequests().

@SpringBootApplication

antMatchers(“/**”).authenticated();

public class SpringBootAdminApplication { // Enable so that the clients can public static void main(String[] args) {

SpringApplication.

authenticate via HTTP basic for registering

http.httpBasic();

run(SpringBootAdminApplication.class, args);

}

}

} // end::conguration-spring-security[]

@Conguration public static class SecurityCong extends

}

WebSecurityCongurerAdapter {

@Override protected void congure(HttpSecurity http)

62

Let us create more Spring Boot applications to monitor through the Spring Boot Admin Server created in the above


Let’s Try Developers artifactId>

1.5.1

org.springframework.boot

spring-boot-starter-actuator
artifactId> Figure 2: Admin server UI

Add the property given below to the application. properties le. This property tells us where the Spring B oot Admin Server is running. Hence, the clients will register with the server. spring.boot.admin.url=http://localhost:1111

Now, if we start the Admin Server and other Spring Boot applications, we will be able to see all the admin clients’ information in the Admin Server dashboard. As we started our Admin Server on port 1111 in this example, we can see the dashboard at http://:1111. Figure 2 shows the Admin Server UI. A detailed view of the application is given in Figure 3. In this view, we can see the tail end of the log le, the metrics, environment variables, the log conguration where we can dynamically switch the log levels at the component level, the root level or package level, and other information. Let’s now look at another feature called notifcations from the Spring Boot admin. This noties the administrators when the application status is DOWN or when the application status is coming UP. Spring Boot admin supports the following channels to notify the user. Email notications Pagerduty notications Hipchat notications Slack notications Let’s Chat notications In this article, we will congure Slack notications. Add the properties given below to the Spring Boot Admin Server’s application.properties le.     

spring.boot.admin.notify.slack.webhook-url=https://hooks. Figure 3: Detailed view of Spring Boot Admin

slack.com/services/T8787879tttr/B5UM0989988L/0000990999VD1hV t7Go1eL //Slack Webhook URL of a channel

steps. All Spring Boot applications that we now create will act as Spring Boot Admin clients. To make the application an admin client, add the dependency given below along with the actuator dependency. In this demo, I have created three applications: Eureka Server, Customer Service and Order Service.

de.codecentric spring-boot-admin-starter-client
spring.boot.admin.notify.slack.message=”*#{application. name}* is *#{to.status}*” //Message to appear in the channel

Since we are managing all the applications with the Spring Boot Admin, we need to secure its UI with a login feature. Let us enable the login feature to the Spring Boot Admin Server. I am going with basic authentication here. Add the Maven dependencies give below to the Admin www.OpenSourceForU.com | OPEN SOURCE FOR YOU | SEPTEMBER 2017 |

63

Developers

Let’s Try

Server module, as follows:

authenticating. Hence, add the properties given below to the admin client’s application.properties fles.

org.springframework.boot

spring.boot.admin.username=admin

spring-boot-starter-security

spring.boot.admin.password=admin123

de.codecentric

spring-boot-admin-server-ui-login
artifactId>

1.5.1

Add the properties given below to the application.properties

There are additional UI features like Hystrix and Turbine UI, which we can enable in the dashboard. You can fnd more details at http://codecentric.github.io/ spring-boot-admin/1.5.1/#_ui_modules . The sample code created for this demonstration is available on https://github. com/2013techsmarts/SpringBoot_Admin_Demo.

fle. security.user.name=admin //user name to authenticate security.user.password=admin123 //Password to authenticate

As we have added security to the Admin Server, admin clients should be able to connect to the server by

Continued from page 60....

By: Siva Prasad Rao Janapati The author is a software engineer with hands-on experience in Java, JEE, Spring, Oracle Commerce, MOZU Commerce, Apache Solr and other open source/enterprise technologies. You can reach him at his blog http://smarttechie.org.

To locate by CSS Selector, type: driver.fndElement(By.cssSelector());

Selenium locators Locator is a command that instructs the Selenium IDE which GUI element it needs to work on. Elements are located in Selenium WebDriver with the help of fndElement() and fndElements() methods provided by the WebDriver and WebElement class. The fndElement() method returns a WebElement object based on a specifed search criteria or ends up throwing an exception. The fndElements() method returns a list of WebElements matching the search criteria. If these are not found, it returns an empty list. The different types of locators are: 1. ID 2. Name 3. Link Text 4. CSS Selector 5. DOM 6. XPath To locate by ID, type: driver.fndElement(By.id());

To locate by name, type:

To locate by XPath, type: driver.fndElement(By.xpath());

Limitations of Selenium Selenium does have some limitations which one needs to be aware of. First and foremost, image based testing is not clearcut compared to some other commercial tools in the market, while the fact that it is open source also means that there is no guaranteed timely support. Another limitation of Selenium is that it supports Web applications; therefore, it is not possible to automate the testing of non-browser based applications. Selenium is a power testing framework to conduct functional and regression testing. It is open source software and supports various programming environments, OSs and popular browsers. Selenium WebDriver is used to conduct batch testing, cross-platform browser testing, data driven testing, etc. It is also very cost-effective when automating Web applications; and for the technically inclined, it provides the power and exibility to extend its capability many times over, making it a very credible alternative to other test automation tools in the market.

driver.fndElement(By.name());

To locate by Link Text, type:

By: Neetesh Mehrotra The author works in TCS as a systems engineer. His areas of interest are Java development and automation testing. You can contact him at [email protected].

driver.fndElement(By.linkText());

64



Splinter: An Easy Way to Test Web Applications Splinter is a Web application testing tool which is built around Python. It automates actions such as visiting specified URLs and interacts with their items. It also removes the drudgery from Web application testing by replacing manual testing with automated testing.

E

very one of us makes mistakes—some of these might be trivial and be ignored, while a few that are serious can’t be ignored. Hence, it’s always a good practice to verify and validate what we do in order to eliminate the possibility of error. So is the case with any software application. The development of a software application is complete only when it’s fully veried and validated (its functionality, performance, user interface, etc). Only then is it ready for release. Carrying out all such validations manually is quite time consuming; so, machines perform such repetitive tasks and processes. This is called automation testing. It saves a lot of time while it reduces the risk of any further error caused by human intervention. There are different automation tools and frameworks available, of which Splinter is one. It lets us automate different manual tasks and processes associated with any Web-based software application. In a Web application, we need to automate the sequence of different actions performed, right from opening the Web browser to checking if it’s loading properly for different actions that involve interactions with the application. Splinter is quite good in automating a sequence of actions. It is an open source tool used for testing different Web applications using Python. The tasks needed to be performed by Splinter are written in Python. It lets us automate various browser actions, such as visiting URLs as well as interacting with their different items. It has got easy-

to-use built-in functions for the most frequently performed tasks. A newbie can easily use Splinter and automate any specic process with just a limited knowledge of Python scripting. It acts as an easily usable abstraction layer on top of different available automation tools like Selenium and makes it easy to write automation tests. We can easily automate a plethora of tasks such as opening a browser, clicking on any specic link or accessing any link, just with one or two lines of code using Splinter, while in the case of other open source tools like Selenium, this is a long and complex process. Splinter even allows us to nd different elements of any Web application using its different properties like tag name, text or ID value, xpath, etc. Since Splinter is an open source tool, it’s quite easy to get clarications on anything that’s not clear. It is supported by a large community. It even has well maintained documentation which makes it easy for any newbie to master this tool. Apart from all this, Splinter supports various inbuilt libraries making the task of automation easier. We can easily manage different actions performed on more than one Web window at the same time as well as navigate through the history of the page, reload the page, etc.

Features of Splinter 1. Splinter has got one of the simplest APIs among open source tools used for automating different tasks on Web applications. This makes it easy to write automated tests


65

Developers

Let’s Try

for any Web application. 2. It supports different Web drivers for various browsers. These drivers are the Firefox Web driver for Mozilla Firefox, Chrome’s Web driver for Google Chrome, PhantomJs Web driver for PhantomJs, zope.testbrowser for Zopetest and a remote Web driver for different ‘headless’ (with no GUI) testing. 3. Splinter also allows us to nd different elements in any Web page by their Xpath, CSS, tag value, name, ID, text or value. In case we need more accurate control of the Web page or we need to do something more, such as interacting with old «frameset» tags, Splinter even exposes the Web driver that allows us to use the low level methods used for interacting with that tag. 4. Splinter supports multiple Web automation back-ends. We can use the same set of test code for doing browser-based testing with Selenium as its back-end, and for ‘headless’ testing with zope.testbrowser as its back-end. 5. It has extensive support for using iframes and interacts with them by just passing the iframe’s name, ID or index value. There is also Chrome support for various alerts and prompts in the Splinter 0.4 version. 6. We can easily execute JavaScript in different drivers which support Splinter. We can even return the result of the script using an inbuilt method called evaluate_script . 7. Splinter has got the ability to work with AJAX and asynchronous JavaScript using various inbuilt methods. 8. When we use Splinter to work with AJAX and asynchronous JavaScript, it’s a common experience to have some elements which are not present in HTML code (since they are created using JavaScript, dynamically). In such cases, we can use various inbuilt methods such as is_element_present or is_text_present for checking the existence of any specic element or text. Splinter will actually load the HTML and the JavaScript in the browser, and the check will be performed before JavaScript is processed. 9. The Splinter project has full documentation for its APIs and this is really important when we have to deal with different third party libraries. 10. We can also easily set up a Splinter development environment. We need to make sure we have some basic development tools in our machine, before setting up an entire environment with just one command. 11. There is also a provision for creating a new Splinter browser in an easy and simple way. We just need to implement a test case for this. 12. Using Splinter, it’s possible to check the HTTP status code that a browser visits. We can use the status_code. is_success method to do the work for us. We can compare the status code directly. 13. Whenever we use the visit method, Splinter actually checks if the given response is a success or not, and if it is not, then Splinter raises an HttpResponseError exception. 66

This helps to conrm if the given response is okay or not. 14. It is pos sible to manipulate the cookies that are us ing the cookies’ attributes from any browser instance. The cookie’s attribute is actually an instance of a CookieManager class which manipulates cookies, such as adding and deleting them. 15. One can create new drivers using Splinter. For instance, if we need to create a new Splinter browser, we just need to implement a test case (extending test.base.BaseBrowsertests). All this will be present in a Python le, which will act as a driver for any future usage.

Chrome

Splinter

Browser-based

Firefox

Selenium Test Code

A P I

Remote

Web Driver

HTTP

Remote Webdriver Server Sauce Labs (IE)

Headless

PhantomJS zope.testbrowser

Figure 1: Flow diagram for Splinter acting as an abstraction layer ( Image source: googleimages.com)

Drivers supported by Splinter Drivers play a signicant role when it comes to any Web application. In Splinter, a Web driver helps us open that specic application whose driver we are using. Different types of drivers are supported by Splinter, based on the way any specic application is accessed and tested. There are browser based drivers, which help to open specic browsers; apart from that we have headless drivers, which help in headless testing and then there are remote drivers, which help to connect to any Web application present on a remote machine. Here is a list of drivers that are supported by Splinter. Browser based drivers:   

Chrome WebDriver Firefox WebDriver Remote WebDriver

Headless drivers     

Chrome WebDriver Phantomjs WebDriver zope.testbrowser Django client Flask client

Remote driver 

Remote WebDriver

Prerequisites and installation of Splinter To install Splinter, Python 2.7 or above should be installed on the system. We can download Python from http://www.python.org.Make sure you have already set up your development environment.


Let’s Try Developers Git should be installed on the system. If you want to use Google Chrome as the Web browser, make sure Chrome WebDriver is set up properly. There are two ways in which we can install Splinter. To install the stable release: To install the ofcial and bug-free version, run the following command from a terminal:

browser.visit(url)

$ [sudo] pip install splinter

#Inbuilt function browser.ll uses the tag name for Email

#checks if Facebook web page is loaded else prints an error message if browser.is_text_present(‘www.facebook.com’): # lls the user’s email ID and password in the email and password eld of the facebook

login section

and Password input box i.e. email and pass respectively to

For installing under-development source-code: To get Splinter’s latest and best features, just run the following given set of commands from a terminal:

identify it browser.ll(‘email’, user_email) browser.ll(‘pass’, user_pass) #selects the login button using its id value present on the

$ git clone git://github.com/cobrateam/splinter.git

Facebook page to click and log in with the given details

$ cd splinter

button = browser.nd_by_id(‘u_0_d’)

$ [sudo] python setup.py install

button.click()

else:

Writing sample code to automate a process using Splinter

print(“Facebook web application NOT FOUND”)

As already stated, even a newbie without much knowledge of programming can automate any specic task using Splinter. Let’s discover how one can easily make Splinter perform any specic task automatically on a Web application. The credit for the ease of coding actually goes to the different inbuilt functions that Splinter possesses. We just need to incorporate all such built-in functions or library les with the help of a few lines of code. Additionally, we need to apply logic while coding to validate different scenarios from different perspectives. Let’s have a look at one of the sample code snippets that has been written for Splinter. Here, we make use of the name and ID values of different elements present on the Web page to identify that specic Web element. Scenario for sample code: Login to a Facebook account using the user’s email ID and password . #imports the Browser library for Splinter from splinter import Browser # takes the email address from user as input to login to his/ her Facebook account user_email = raw_input(“enter users email address “) # takes the password from user as input to login to his/her Facebook account user_pass = raw_input(“enter users password “)

Some important built-in functions used in Splinter Table 1 lists some of Splinter’s significant built-in functions that can be used while automating any process for a Web application.

Setting up the Splinter development environment When it comes to programming in Splinter, we have already seen that it’s easier than other open source Web application testing tools. But we need to set up a development environment for it, wherein we can easily code or automate a specic process using Splinter. This is not a tough task. We just need to make sure that we have some basic development tools, library les and a few add-on dependencies on our machine, which will ultimately help us code in an easier and better way. We can get the required tools and set up the entire environment using just a few commands. Lets’ have a look at the different development tools required to set up the environment. Basic development tools: If you are using the Mac OS, install the Xcode tool. It can be downloaded from the Mac Application Store (on the Mac OS X Lion) or even from the Apple website. If you are using a Linux computer, install some of the basic development libraries and the headers. On Ubuntu, you can easily install all of these using the apt-get command. Given below is the command used for this purpose.

# loads the Firefox browser

$ [sudo] apt-get install build-essential python-dev libxml2-

browser= Browser(‘refox’)

dev libxslt1-dev

# stores the URL for Facebook in url variable url = “https://www.facebook.com/” #navigates to facebook website and load that in the Firefox browser

Pip and virtualenv: First of all, we need to make sure that we have Pip installed in our system, with which we manage all the Splinter development dependencies. It lets us program our task and makes the system perform any activity www.OpenSourceForU.com | OPEN SOURCE FOR YOU | SEPTEMBER 2017 |

67

Developers

Let’s Try

Table 1

Name of the function Browser Browser () related browser.visit()

For managing different browser windows actions

For finding elements of any Web page

Function used for Used to instantiate any browser and create a window for it This function is used to navigate to any specific URL browser.reload() Used to reload any Web page browser.title Displays the title of the current active Web page browser.html Used to display the HTML content of the current active Web page browser.url Used to access the URL of the current active Web page browser.windows[0] Used to access the first window

browser.windows [window_name]

Used to access any specific window using the window_name

browser.reload() browser.title browser.html browser.url browser.windows[numeric value representing window to be visited]

browser.windows[window_name]

browser.windows. Takes you to the current window current () window.is_current () Boolean – used to check whether the current window is active or not

browser.windows.current()

window.next () window.prev() window.close() window.close_oth- ers() browser.find_by_ name()

Takes you to the next open window Takes you to the previous open window Closes current window Closes all windows except the current one Used to find an element using its name

window.next() window.prev() window.close() window.close_others()

browser.find_by_ css() browser.find_by_ xpath() browser.find_by_ tag() browser.find_by_ text() browser.find_by_id()

Used to find an element using i ts CSS value Used to find an element using its XPath

browser.find_by_ value()

Used to find an element using its value

Used to find an element using i ts tag name Used to find an element using i ts text value Used to find an element using its ID value

using the code or command we write. It’s advisable to choose virtualenv for a good development environment. Once we have all the development libraries installed for the OS we are using, we just need to install all the Splinter development dependencies using the make command. Given below is the command for this.

window.is_current = Boolean True or False

browser.find_by_name(‘name of element’) browser.find_by_css(‘css value’) browser.find_by_xpath(‘xpath value’) browser.find_by_tag(‘name of tag’) browser.find_by_text(‘text value for the element to be accessed’) browser.find_by_id(‘id value of the element’) browser.find_by_value(‘value of the element to be accessed’)

References [1] http://www.wikipedia.org/ [2] https://splinter.readthedocs.io [3] https://github.com/cobrateam/splinter

$ [sudo] make dependencies

We will use sudo while making dependencies only if we are not using virtualenv.

68

Syntax of the function variable name = Browser (‘name of the Web driver used’) browser.visit(‘URL’)


By: Vivek Ratan The author, who is currently an automation test engineer at Infosys, Pune, has completed his B. Tech in electronics and instrumentation engineering. He can be reached at [email protected] for any suggestions or queries.


Getting Started with PHP, the Popular Programming Language This article provides an introduction to PHP — the development process, its history, as well as its pros and cons. At the end of the article, we learn how to install XAMPP on a computer, and to write code to add two numbers.

I

f you refer to any Web technology survey to check the market share of different server side scripting languages, you will be surprised to know that PHP is used by an average 70 per cent of the websites. According to w3techs. com, “PHP is used by 82.7 per cent of all the websites whose server-side programming language we know.” In the early stages, even Facebook servers deployed PHP to run their social networking application. Nevertheless, we are not concerned about the Web trafc hosted by PHP these days. Instead, we will delve deep into PHP to understand its development, its history, its pros and cons and, in the end, we will have a sneak peek into some of the open s ource IDEs which you can use for rapid development. First, let’s understand what PHP is. It is an abbreviated form of ‘Hypertext Pre-processor’. Confused about the sequence of the acronym? Actually, the earlier name of PHP was ‘Personal Home Page’ and hence the acronym. It is a server side programming language mainly used to enhance the look and feel of HTML Web pages. A sample PHP code embedded into HTML looks like what follows:

Example

In the above example, you can see how easily PHP can be embedded inside HTML code just by enclosing it inside tags, which allows very cool navigation between HTML and PHP code. It differs from client-side scripting languages like JavaScript in that PHP code is executed on the server with the help of a PHP interpreter, and only the resultant HTML is sent to the requester’s computer. Though it can do a variety of tasks, ranging from creating www.OpenSourceForU.com | OPEN SOURCE FOR YOU | SEPTEMBER 2017 |

69

Developers

Let’s Try

forms to generating dynamic Web content to sending and receiving cookies, yet there are three main areas where PHP scripts are usually deployed. Server-side scripting: This is the main usage and target area of PHP. You require a PHP parser, a Web browser and a Web server to make use of it, and then you will be able to view the PHP output of Web pages on your machine’s browser. Command line scripting: PHP scripts can also be run without any server or browser but with the help of a PHP parser. This is most suited for tasks that take a lot of time — for example, sending newsletters to thousands of records, taking backups from databases, and transferring heavy les from one location to another. Creating desktop applications: PHP can also be used to develop desktop based applications with graphical user interfaces (GUI). Though it has a lot of pain points, you can use PHP-GTK for that, if you want to. PHP-GTK is available as an extension to PHP. 



called ‘Personal Home Page Tools’ in order to maintain his personal Web pages. The succeeding year, these tools were released under the name of ‘Personal Home Page/ Forms Interpreter’ as CGI binaries. They were enabled to provide support for databases and Web forms. Once they were released to the whole world, PHP underwent a series of developments and modications, and the result was that the second version of ‘Personal Home Page/Forms Interpreter’ was released in November 1997. Moving on, PHP 3, 4 and 5 were released in 1998, 2000 and 2004, respectively. Today, the most used version of PHP is PHP 5, with approximately 93 per cent of the websites using PHP making use of it, though PHP 7 is also available in the market. In 2010, PHP 5.4 came out with Unicode support added to it.



Fact: Did

you know that PHP has a mascot just like sports teams? The PHP mascot is a big blue elephant named elePHPant.

PHP and HTML – similar but different PHP is often confused with HTML. So to set things straight, let’s take a look at how PHP and HTML are different and similar at the same time. As we all know, HTML is a markup language and is the backbone for front-end Web pages. On the other hand, PHP works in the background, on the server, where HTML is deployed to perform tasks. Together, they are used to make Web pages dynamic. For better understanding, let’s look at an example where you display some content on a Web page using HTML. Now, if you want to do some backend validation on the database, then you will use PHP to do it. So both HTML and PHP have different assigned roles and they complement each other perfectly. Listed below are some of the similarities and differences that will make this clear.

The pros and cons of PHP Before going further into PHP development, let’s take a look at some of the advantages and disadvantages of using it in Web development. Advantages Availability: The biggest advantage of PHP is that it is available as open source, due to which one can nd a large developer community for support and help. Stability: PHP has been in use since 1995 and thus it’s quite stable compared to other server side scripting languages since its source code is open and if any bug is found, it can be readily xed. Extensive libraries: There are thousands of libraries available which enhance the abilities of PHP—for example, PDFs, graphs, Flash movies, etc. PHP makes use of modules, so you don’t have to write everything from the beginning. You just need to add the required module in your code and you are good to go. Built-in modules: Using PHP, one can connect to the database effortlessly using its built-in modules, which drastically reduce the development time and effort of Web developers. Cross-platform: PHP is supported on all platforms, so you don’t have to worry whether your code written in Windows OS will work on Linux or not. Easy to use: For beginners, learning PHP is easy because of its cool syntax, which is somewhat similar to the C programming language, making it even simpler for those familiar with C. 









Similarities

Differences

Compatible with most of the browsers supporting their technologies.

HTML is used on the front-end whereas PHP is back-end technology.

Can be used on all operating systems.

PHP is a programming language, whereas HTML is called a markup language and is not included in the category of programming languages because it can’t do calculations like ‘1+1=2’.

History and development The development of PHP dates back to 1994 when a DanishCanadian programmer Rasmus Lerdorf created Perl scripts

70



Disadvantages Not suitable for huge applications: Though PHP has a lot of advantages in Web page development, it still can’t be used to build complicated and huge Web applications since it does not support modularity and, hence, the maintenance of the app will be a cumbersome task. Security: Security of data involved in Web pages is of paramount concern. The security of PHP can be 




Let’s Try Developers compromised due to its open source nature since anyone can view its source code and detect bugs in it. So you have to take extra measures to ensure the security of your Web page if you are dealing with sensitive data. Fact: It is estimated that there are approximately 5 million PHP developers worldwide, which is a testament to its power.

Open source IDEs for PHP development The choice of IDE plays an important role in the development of any program or application but this aspect is often neglected. A good and robust IDE comes packed with loads of features and packages to enable rapid development. Automatic code generation, refactoring, organising imports, debugging, identifying dead code and indentation are some of the advantages a powerful IDE can provide. So let’s take a look at some dominant open source IDEs that can be very useful in PHP development. 1. NetBeans: Most of you must be aware of NetBeans in Java development but it can also be used for PHP development. The biggest advantage of NetBeans is that it supports many languages like English, C hinese, Japanese, etc, and can be installed smoothly on any operating system. Some of the features that differentiate it from the rest are smart code completion, refactoring, try/catch code completion and formatting. It also has the capability to congure various PHP frameworks like Smarty, Doctrine, etc. You can download it from netbeans.org. 2. Eclipse: Eclipse tops the list of popular IDEs. If you have worked with Eclipse earlier, then you will feel at home using Eclipse PDT for PHP development. It can be downloaded from eclipse.org/pdt . Some of its features are syntax highlighting, debugging, code templates, validating syntax and easy code management through Windows Explorer. It is a cross-platform IDE and works on Windows, Linux and Mac OS. Since it is developed in Java, you must have it installed in your machine. 3. PHPStorm: PHPStorm, developed by JetBrains (the same company that developed IntelliJ IDEA for Java), is mainly used for professional purposes but is also available licence-free for students, teachers and open source projects. It has the most up-to-date set of features for rapid development since it provides support for leading frontend technologies like HTML5, CofeeScript, JavaScript and Sass. It supports all the major frameworks available in the market like Symfony, CakePHP, Laravel and Zend, and can also be integrated with databases, version control software, rest clients and command line tools to ease the work of developers. A number of MNCs, like Wikipedia, Yahoo and Cisco, are making use of PHPStorm for PHP development. You can read more about PHPStorm at jetbrains.com/phpstorm. 4. Sublime Text: Sublime Text is basically a text editor, but it can be converted into a PHP IDE by installing various

available packages. It is known for its sleek, featurerich and lightweight interface. It is also supported on all operating systems. Some of the packages which can be used to convert it into an IDE are Sublime PHP Companion, PHPCS, codIntel, PHPDoc, Simple PHPunit, etc. It can be downloaded as open source from sublimetext.com. 5. PHP Designer: This IDE is only available for Windows users. It is very fast and powerful, with full support for PHP, HTML, JavaScript and CSS. It is used for fast Web development due to its features like intelligent syntax highlighting, object-oriented programming, code templates, code tips and debug manager, which are all wrapped into a sleek and intuitive interface that can also be customised according to various available themes. It also supports various JavaScript frameworks such as JQuery, ExtJs and Yui. An open source version of it is available and you can read more about it on its ofcial website. 6. NuSphere PHP IDE: PHpED is the IDE developed by NuSphere, a Nevada based company which entered the market way back in 2001. The current available version of PHpED is 18.0 which provides support for PHP 7.0 and almost all PHP frameworks. This tool also has the ability to run unit tests for the developed projects and comes packaged with the support for all Web based technologies. You can download PHpED from NuSphere’s website www.nusphere.com. 7. Codelobster: Codelobster also provides a free IDE for PHP development. Though it is not used too often, it is catching up fast. By downloading the free version, you get support for PHP, JS, HTML and CSS. It can be integrated with various frameworks such as Drupal, WordPress, Symfony and Yii. You can download it from www.codelobster.com.

Writing the first PHP program Having read about PHP, its history and various IDEs, let’s write our rst PHP program and run it using XAMPP. Though there is no ofcial information about the full form of XAMPP, it is usually assumed to stand for cross-platform (X), Apache (A), MariaDB (M), PHP (P) and Perl (P). XAMPP is an open source, widely used Web server developed by apachefriends.org, which can be used to create a local HTTP server on machines with a few clicks. We will also be using it in our tutorial below.

Figure 1: Apache service started on XAMPP control panel


71

Developers 







Let’s Try

Download the latest version of XAMPP on your systems from the website https://www.apachefriends.org/ download.html. After the download is complete, install it in your machine. By default, XAMPP is installed in your machine’s C drive, but if you have specied any other directory in the installation process, go to that directory and create a folder named PHPDevelopment inside the htdocs folder in your XAMPP installation directory folder. For example, C:\xampp\htdocs\PHPDevelopment. Now start the XAMPP control panel and click on the Start button to start Apache. Create a text le inside the above folder named AddTwoNumbers.php and copy the following code inside it:

Index of/PHPDevelopment Parent Directory Add Two Numbers.php 2017-07-30 13:47 545 FirstDemo.php 2017-07-30 13:24 333 Apache/2.4.26 (Win32) OpenSSL/1.0.21 PHP/5.6.31 Server Figure 2: Screenshot of the list of files in your directory

Addition of Two Number

Sum

Addition Of Two Numbers

Number 1:

Figure 3: PHP program when run on the browser

Number 2:



Now type localhost/ PHPDevelopment and it will list all the files in your directory on your browser, as shown in Figure 2. Click on AddTwoNumbers.php and you will be directed to the required page, where you can perform the addition of two numbers. Here, you can see that the form has been created using HTML and the corresponding addition of the numbers is done using PHP. Now start your Web development using PHP. You can also make use of the various frameworks available to simplify development and lessen your coding time. Happy coding! 

$num1 and $num2 is $sum”;

}

By: Vinayak Vaid

?>

The author works as an automation engineer at Infosys Limited, Pune. He has worked on different testing technologies and automation tools like QTP, Selenium and Coded UI. He can be contacted at [email protected].




72


How To

Developers

Crawling the Web with Scrapy Web crawling or spidering is the process of systematically extracting data from a website using a Web crawler, spider or robot. A Web scraper methodically harvests data from a website. This article takes the reader through the Web scraping process using Scrapy.

S

crapy is one of the most powerful and popular Python frameworks for crawling websites and extracting structured data useful for applications like data analysis, historical archival, knowledge processing, etc. To work with Scrapy, you need to have Python installed on your system. Python can be downloaded from www.python.org.

following le structure: “scrapy_frst/

-scrapy.cfg

scrapy_frst/

-__init__.py - items.py

Installing Scrapy with Pip

-pipelines.py

Pip is installed along with Python in the Python/Scripts/folder. To install Scrapy, type the following command:

-settings.py

-spiders/

-__init__.py”

pip install scrapy

The above command will install Scrapy on your machine in the Python/Lib/site-packages folder.

Creating a project With Scrapy installed, navigate to the folder in which you want to create your project, open cmd and type the command below to create the Scrapy project: scrapy startproject scrapy_frst

The above command will create a Scrapy project with the

In the folder structure given above, ‘scrapy_rst’ is the root directory of our Scrapy project. A spider is a class that describes how a website will be scraped, how it will be crawled and how data will be extracted from it. The customisation needed to crawl and parse Web pages is dened in the spiders.

A spiders’s scraping life cycle 1. You start by generating the initial request to crawl the rst URL obtained by the start_requests() method, which generates a request for the URLs specied in the start_urls, and parses them using the parse method as a www.OpenSourceForU.com | OPEN SOURCE FOR YOU | SEPTEMBER 2017 |

73

Developers

How To

Scrapy commands scrapy startproject myproject

[project_dir] scrapy genspider spider_name

[domain.com] Scrapy bench scrapy check

This command will create a Scrapy project in the project directory specified; else with the name of project , if project_dir is not mentioned. This command needs to be run from the root directory of the project, to create a spider with allowed_domain as domain.com. This runs a quick benchmark test, to tell you Scrapy’s maximum possible speed in crawling Web pages, given your hardware. Checks spider contracts.

scrapy crawl [spider]

This command instructs the spider to start crawling the Web pages.

scrapy edit [spider]

This command is used to edit the spider using the editor specified in the EDITOR environment variables or EDITOR setting.

scrapy fetch [url] scrapy list scrapy parse [url]

This command downloads the contents of the URL and stores them in a standard output file. Lists the available spiders in the project. This is the default callback used by Scrapy to process downloaded responses, when their requests don’t specify a callback.

scrapy runspider le_name.py

Runs a spider self-contained in a Python file without having to create a project.

scrapy view [url]

Opens the URL in the browser as seen by the spider.

scrapy settings

This is to get the Scrapy setting value.

scrapy version

This is to get the Scrapy version installed.

scrapy shell [url optional]

Opens the interactive Scrapy console for the URL.

callback to get a response. 2. In the callback, after the parsing is done, either of the three dicts of content — request object, item object or iterable — is returned. This request will also contain the callback and is downloaded by Scrapy. The response is handled by the corresponding callback. 3. In callbacks, parsing of page content is performed using the XPath selectors or any other parser libraries like lxml, and items are generated with parsed data. 4. The returned items are then persisted into the database or the item pipeline, or written to a le using the FeedExports service. Scrapy is bundled with three kinds of spiders . BaseSpider: All the spiders must inherit this spider. It is the simplest one, responsible for start_urls / start_request() and calling of the parse method for each resulting response. CrawlSpider: This provides a convenient method for crawling links by dening a set of rules. It can be overridden as per the project’s needs. It supports all the BaseSpider’s attributes as well as an additional attribute, ‘rules’, which is a list of one or more rules. XMLSpider and CSVSpider: XMLSpider iterates over the XML feeds through a certain node name, whereas C SVSpider is used to crawl CSV feed. The difference between them is

74

that XMLSpider iterates over nodes and CSVSpider iterates over rows with the parse_rows() method. Having understood the different types of spiders, we are ready to start writing our rst spider. Create a le named myFirstSpider.py in the spiders folder of our project. import scrapy class MyrstspiderSpider(scrapy.Spider): name = “myFirstSpider” allowed_domains = [“opensourceforyou.com”] start_urls = (

‘http://opensourceforu.com/2015/10/building-a-

django-app/’, ) def parse(self, response): page = response.url.split(“/”)[-2] lename = ‘quotes-%s.html’ % page with open(lename, ‘wb’) as f:

f.write(response.body) self.log(‘Saved le %s’ % lename)

In the above code, the following attributes have been dened: 1. name: This is the unique name given to the spider in the project.


How To

2. allowed_domains: This is the base address of the URLs that the spider is allowed to crawl. 3. Start_requests(): The spider begins to crawl on the requests returned by this method. It is called when the spider is opened for scraping. 4. Parse(): This handles the responses downloaded for each request made. It is responsible for processing the response and returning scraped data. In the above code, the parse method will be used to save the response.body into the HTML le.

Developers

dene the item Fields in our project’s items.py le. Add the following lines to it: title=item.Field() url=item.Field()

Our code will look like what’s shown in Figure 1.

Crawling Crawling is basically following links and crawling around websites. With Scrapy, we can crawl on any website using a spider with the following command: scrapy crawl myFirstSpider Figure 1: First item

Extraction with selectors and items Selectors: A certain part of HTML Source can be scraped using selectors, which is achieved using CSS or Xpath expressions. Xpath is a language for selecting nodes in XML documents as well as with HTML, whereas CSS selectors are used to dene selectors for associate styles. Include the following code to our previous spider code to select the title of the Web page:

After the changes in the item are done, we need to make some changes in our spider. Add the following lines so that it can yield the item data: from scrapy_rst.items import ScrapyFirstItem def parse(self, response):

item=ScrapyFirstItem()

item[‘url’]=response.url

def parse(self, response):

url=response.url

for select in response.xpath(‘//title’):

for select in response.xpath(‘//title’):

title=select.xpath(‘text()’).extract() self.log(“title here %s” %title)

Items: Items are used to collect the scraped data. They are regular Python dicts. Before using an item we need to

title=select.xpath(‘text()’).extract() self.log(“title here %s” %title)

item[‘title’]=title yield item

Now run the spider and our output will look like what’s shown in Figure 2.

Figure 2: Execution of the spider


75

Developers

How To

Scraped data

gethostname())”

After data is scraped from different sources, it can be persisted into a le using FeedExports, which ensures that data is stored properly with multiple serialisation formats. We will store the data as XML. Run the following command to store the data:

Increment stat value: “stats.inc_value(‘count_variable’)”

scrapy crawl myFirstSpider -o data.xml

We can nd data.xml in our project’s root folder, as shown in Figure 3.

Get stat value: “stats.get_stats()”

3. Sending email: Scrapy comes with an easy-to-use service for sending email and is implemented using a twisted nonblocking IO of the crawler. For example: from scrapy.mail import MailSender

mailer=MailSender()

mailer.send(to=[‘[email protected]’],subject=”Test Subject ” ,body=”Test Body”, cc=[‘[email protected]’])

4. Telnet console: All the running processes of Scrapy are controlled and inspected using this console. It comes enabled by default and can be accessed using the following command: Figure 3: data.xml telnet console 6023

5. Web services: This service is used to control Scrapy’s Web crawler via the JSON-RPC 2.0 protocol. It needs to be installed separately using the following command: Figure 4: Final spider pip install scrapy-jsonrpc

Our nal spider will look like what’s shown in Figure 4.

Built-in services 1. Logging: Scrapy uses Python’s built-in logging system for event tracking. It allows us to include our own messages along with third party APIs’ logging messages in our application’s log.

The following lines should be included in our project’s settings. py le: EXTENSIONS={‘scrapy_jsonrpc.webservice.WebService’:500,} Set JSONRPC_ENABLED settings to True.

Scrapy vs BeautifulSoup import logging logging.WARNING(‘this is a warning’) logging.log(logging.WARNING,”Warning Message”) logging.error(“error goes here”) logging.critical(“critical message goes here”) logging.info(“info goes here”) logging.debug(“debug goes here”)

2. stats collection: This facilitates the collection of stats in a key value pair, where values are often counter. This service is always available even if it is disabled, in which case the API will be called but will not collect anything. The stats collector can be accessed using the stats attribute. For example:

Scrapy: Scrapy is a full-edged spider library, capable of performing load balancing restrictions, and parsing a wide range of data types with minimal customisation. It is a Web scraping framework and can be used to crawl numerous URLs by providing constraints. It is best suited in situations like when you have proper seed URLs. Scrapy supports both CSS selectors and XPath expressions for data extraction. In fact, you could even use BeautifulSoup or PyQuery as the data extraction mechanism in your Scrapy spiders. BeautifulSoup: This is a parsing library which provides easy-tounderstand methods for navigating, searching and nally extracting the data you need, i.e., it helps us to navigate through HTML and can be used to fetch data and parse it into any specic format. It can be used if you’d rather implement the HTML fetching part yourself and want to easily navigate through HTML DOM.

class ExtensionThatAccessStats(object): def __init__(self,stats):

https://doc.scrapy.org

self.stats=stats @classmethod

By: Shubham Sharma

def from_crawler(cls,crawler):

The author is an open source activist working as a software engineer at KPIT Technologies, Pune. He can be contacted at [email protected].

return cls(crawlse.stats) Set stat value: “stats.set_value(‘hostname’,socket.

76

Reference


Overview

Developers

Five Friendly Open Source Tools for Testing Web Applications Web application testing helps ensure that the apps conform to certain set standards. It is a means to check for any bugs in the application before the latter goes live or the code is released on the Internet. Various aspects of the application and its behaviour under different conditions are checked. Here’s a brief introduction to five popular open source tools you can use for this job.

T

he term ‘Web application’ or ‘Web app’ is often confused with ‘website’. So let’s get that doubt cleared —a Web application is a computer app that is hosted on a website. A website has some xed content while a Web application performs various denite actions based on the users’ inputs and actions.

Types of testing 1. Functional testing: Functional testing is a superset validating all those features and functionalities that the application is meant to perform. It includes testing the business logic around the set rules. Listed below are some of the common checkpoints: Tests links to a page from external pages. Validates the response to a form submission. Checks, creates, reads, updates, deletes (CRUD) tasks. Veries that the data retrieved is correct. Identies database connectivity and query errors. 2. Browser compatibility testing: Because of the availability of cross-platform browser versions, it 

Web application testing Web application testing involves all those activities that software testers perform to certify a Web app. This testing has its own set of criteria and checkpoints, based on the development model, to decide whether the actions are part of expected behaviour or not.

   


77

Developers Overview has become necessary to validate if the application is supported on other browser versions without compatibility issues. If the application is not behaving properly on certain browsers, it is good to mention the supported versions to avoid customer complaints. Below are some of the common checkpoints: Checks browser rendering of your application’s user interface. Checks the browser’s security settings for cross-domain access and hacks. Veries consistent functioning of the app across multiple versions of a browser. Checks user interface rendering on different-sized mobile device screens, including screen rotation. Veries that the application operates correctly when the device moves in and out of the range of network services. 3. Performance testing: Performance testing focuses on checking how an application behaves under extra load, which refers to the number of users accessing the application simultaneously. It is good to see which particular feature is breaking down under the given load. Listed below are some of the common checkpoints: Checks the server’s response to the browser form submit requests. Identies changes in performance over a period of time. Tests for functions that stop working at higher loads. Identies how an application functions after a system crash or component failure. Identies forms and links that operate differently under higher loads. 4. Security testing: Securing user data is a critical task and Web apps should not leak data. Testing ensures that the app works only with a valid login, and that after logout, the data remains secure and pressing the ‘back’ key does not resume the session. Given below are some of the common checkpoints: Checks whether the app operates on certain URLs without logging. Tests basic authentication using false user names and password credentials. Tests if the app functions correctly upon invalid URL attribute values. Checks how the app functions with invalid input elds, including text elds. Tests CAPTCHA elds for Web forms and logins. 5. Usability testing: Any Web app is considered user friendly if accessibility is easy and navigation is smooth. If there are ambiguities in representations, then these should be corrected. Users want clear descriptions and representations. Shown below are some of the common checkpoints: Tests that the content is logically arranged and easy for users to understand. Checks for spelling errors. Checks that pages adhere to colour and pattern style  









 

guidelines, including fonts, frames and borders. Checks that images load correctly and in their proper size. With the increasing need to analyse the performance of your Web app, it is a good idea to evaluate some of the popular open source performance testing tools. 

Why choose open source performance test tools? 1. No licensing costs – a commercial load testing tool can really burn a hole in your pocket when you want to test with a large number of virtual users. 2. Generates (almost) an innite amount of load on the Web app without charging users any additional licensing costs. The only limitation would be the resources available. 3. Enables you to create your own plugins to extend the analysis and reporting capabilities. 4. Integrates with other open source and commercial tools to drive end-to-end test cycles.

Popular open source Web application test tools Licensed tools have their own benets but open source always stands out because of the ease of use. Here are some popular open source Web app test tools that are easily available and simple to use as well.



1. JMeter: Load and performance tester JMeter is a pure Java desktop application designed to load-test functional behaviour and measure performance. It can be used to test performance both on static and dynamic resources (les, Servlets, Perl scripts, Java objects, databases and queries, FTP servers and more). It can be used to simulate a heavy load on a server, network or object to test its strength or to analyse the overall performance under different load types. JMeter was originally used for testing Web and FTP applications. Nowadays, it is used for functional tests, database server tests, etc.











The pros of JMeter A very lightweight tool that can be installed easily. As it is an open source tool, you need not be worried about the licence. There are multiple plugins that are available in the market  







 

78

Figure 1: JMeter


Overview



and can be installed easily, according to requirements. Offers caching and ofine analysis/replaying of test results.



The cons of JMeter It can be used only on Web applications. Consumption of memory is high in GUI mode, and load, stress and endurance testing with high user loads should preferably be run in non-GUI mode. Complex scenarios cannot be checked using JMeter thread group. Recording is complex, as we need to set up the proxy manually. It supports only Java for custom coding.

Developers

Capybara uses the same DSL to drive a variety of browsers and headless drivers.

The cons of Capybara The only con of this tool is that its framework adds a layer on top of the actual implementation which makes it tough to debug what is actually happening.







3. Selenium: Web app testing tool



Selenium is a suite of tools such as Selenium IDE, Selenium Remote Control and Selenium Grid to test the Web application. Selenium IDE is an integrated development environment for Selenium scripts. It is implemented as a Firefox extension, and allows you to record, edit, and debug tests. It supports record and playback.





2. Capybara: Acceptance test framework for Web applications Capybara is a Web based automation framework used for creating functional tests that simulate how users interact with your application. It is a library built to be used on t op of an underl ying Web based driver. It offers a user friendly DSL (domain specific language), which is used to describe actions that are executed by the underlying Web driver. When the page is loaded using the DSL (and underlying Web driver), Capybara will try to locate the relevant element in the DOM (Document Object Model) and execute the action, such as click a button, link, etc.

The pros of Selenium It is a low cost tool. It can carry out browser compatibility testing. It offers a choice of languages. It has multiple testing frameworks. It is easy to integrate with the testing ecosystem. It is open for enhancement. It has test-driven development. It’s useful for comprehensive testing.        



The cons of Selenium There are a few problems while testing. There are issues with nding locators. There are limitations in browser support. Manual scripts are not allowed. The performance is slow.



4. Sahi: An automation and testing tool



The pros of Capybara No set-up necessary for Rails and Rack applications. It works out-of-the-box. Intuitive API, which mimics the language an actual user would use. Powerful synchronisation features mean you never have to manually wait for asynchronous processes to complete. 

Figure 2: Selenium’s capabilities

   

Sahi is an automation and testing tool for Web applications. It is available in both open source and proprietary versions. The open source version includes record and playback on all browsers, HTML reports, suites and batch run, and parallel playback. The Pro version includes some

Figure 3: Sahi’s interface


79

Developers Overview Table 1: A comparison of open source Web app testing tools

Title

Company

Scope

Application rights User interface available Supported technology

JMeter

Apache

Test automation framework; Testing tool

Free use, open source

SeleniumHQ

GitHub project, Test automation Free use, Google Code framework; open source Projects, Testing tool SeleniumHQ

Integrated into ALM, Maven; standalone application; Web based

Adobe Flash, Ajax, .NET, DOM, Java GUI, Android apps, Silverlight, CSS, HTML, HTTP, Xpath

Capybara

GitHub project


Free use, open source

COM API; Tool extension; Web based

Web, Web services

Sahi Pro

Tyto Software


Commercial, trial

Command line

Adobe Flex, Ajax, Java, PHP, RubyOnRails, HTTPS, JavaScript

WebLOAD

RadView Software

Cloud services; Testing tool

Commercial, demo, free use, trial

Batch mode, plugin, stan- JDBC driver, JMS, dalone application LDAP CORBA, IMAP, POP3, SMTP, SOAP, Web, HTTP, HTTPS

of the enhanced features like test distribution and report customisation. Sahi runs as a proxy server; the proxy settings are congured to point to Sahi’s proxy and then inject JavaScript event handlers into Web pages. The pros of Sahi Sahi can achieve most of the automation with the available functions and variables. It has all the inbuilt APIs required for complex tasks. Sahi also has multibrowser support. It does not require additional tools to run and execute the tests. All the tests run from the inbuilt Sahi Controller.

Mobile applications, Web

and a sophisticated analytics dashboard. WebLOAD has built-in exibility, allowing QA and DevOps teams to create complex load testing scenarios thanks to native Java scripting. WebLOAD supports hundreds of technologies – from Web protocols and enterprise applications to network and server technologies.





The cons of Sahi Compared to Selenium, Sahi is difcult to start as it involves a complex Figure 4: WebLOAD installation process. It also has a very confusing interface. It does not provide the same visibility that Selenium does, is less popular and has the smallest and least developed community. 

The pros of WebLOAD It has native JavaScript scripting. UI wizards enhance the script. It supports many technologies. It offers easy-to-reach customer support.    

The cons of WebLOAD It does not support Citrix. It does not support the SAP GUI. It does not support RDP and RTE. Table 1 compares the merits of all the testing solutions.   



5. WebLOAD: The best load-runner alternative WebLOAD is an enterprise-scale load testing tool which features a comprehensive IDE, a load generation console,


By: Meghraj Singh Beniwal The author has a B.Tech in electronics and communications, and is currently working as an automation engineer in a company located in Virginia (USA). He can be contacted at meghrajsingh01@ rediffmail.com or [email protected].



80


Insight Developers

Developing Research Based Web Applications Using Red Hat OpenShift

OpenShift is a Kubernetes based container application, designed for co ntainer based software deployment and management. It is an application development and hosting platform, which automates management and enables the developer to focus on the app itself.

T

he increase in the volume, velocity and the variety of data from multiple channels demands high performance computing resources that can process heterogeneous Big Data. It is not always possible to purchase costly computing resources like high performance multicore processors with supercomputing powers, huge memory devices and related technologies to process, visualise and make predictions on the datasets related to live streaming and real-time supercomputing applications. To cope with and work with such technologies, cloud services are used, whereby computing resources can be hired on demand and billed for as per usage. There are a number of cloud services providers in the global market with different delivery models including Infrastructure-as-a-Service (IaaS), Platform-as-a-Service (PaaS) and Software-as-a-Service (SaaS). Nowadays, there are some new keywords in the cloud delivery space, like Network-as-a-Service (NaaS), Database-as-a-Service (DBaaS), Testing-as-a-Service (TaaS) and many others. Each of these cloud delivery approaches has different resources, which are used for different applications.

Features of Red Hat OpenShift Red Hat OpenShift is one of the leading cloud services providers in the PaaS (Platform as a Service) paradigm. It provides multiple platforms to cloud users with the exibility to develop, deploy and execute applications on the cloud. OpenShift has high performance data centres with enormous processing power to work with different programming languages, which include Java, PHP, Ruby, Python, Node.js, Perl, Jenkins Server, Ghost, Go and many others. A beginner can use the Free Tier of Red Hat OpenShift for the development, deployment and execution of new cloud apps on the online platform provided by it. Any of the programming languages mentioned can be used for the development of apps with real-time implementation.

Developing PHP research based Web applications OpenShift provides multiple programming language options to cloud users for the development of apps. With each programming language, OpenShift delivers multiple versions so that the compatibility issues can be avoided at later stages. www.OpenSourceForU.com | OPEN SOURCE FOR YOU | SEPTEMBER 2017 |

81

Developers Insight

Figure 1: The OpenShift portal

Figure 2: Login panel for Red Hat OpenShift Figure 5: Selecting the programming language on OpenShift

Figure 6: Assigning the URL to the PHP application on O penShift Figure 3: Starter and pro plans for cloud users on OpenShift

the commands that should be executed on the local command prompt or in the Linux Shell.

Developing a Twitter extraction using PHP on the OpenShift PaaS To develop a Twitter extraction on the OpenShift PaaS, use the code given below:
$settings = array( ‘oauth_access_token’ => “XXXXXXXXXXXXXXXXXXXXXXXXX”,

The cloud applications can be uploaded using mapping with GIT via a local command prompt (Windows CMD or Linux Terminal). OpenShift specifies

82

‘oauth_access_token_secret’ => “ XXXXXXXXXXXXXXXXXXXXXXXXX “,


‘consumer_key’ => “ XXXXXXXXXXXXXXXXXXXXXXXXX “,

Insight Developers

Figure 9: Copying local files to the OpenShift cloud Figure 7: Selecting the region the cloud application should be deployed in

Figure 10: Committing the changes as a permanent write operation on the cloud Figure 8: Mapping of the command prompt with GIT to upload t he code on the live cloud ‘consumer_secret’ => “ XXXXXXXXXXXXXXXXXXXXXXXXX “ ); $twitterurl = “https://api.twitter.com/1.1/statuses/user_ timeline.json”; $requestMethod = “GET”; if (isset($_GET[‘user’])) {$user = $_GET[‘user’];} else {$user = “gauravkumarin”;} if (isset($_GET[‘count’])) {$count = $_GET[‘count’];} else

Figure 11: View the configuration and URL of the cloud app

{$count = 20;} $feld = “?screen_name=$user&count=$count”; $mytwitter = new TwitterAPIExchange($settings); $str = json_decode($mytwitter->setGetfeld($feld) ->buildOauth($twitterurl, $requestMethod) ->performRequest(),$assoc = TRUE); if($str[“errors”][0][“message”] != “”) {echo “
Sorry, there was a problem.
Twitter returned the following error message:
”.$str[errors][0][“message”].”
”;exit();} foreach($str as $current) Figure 12: Executing the Twitter timeline extraction on the OpenShift cloud

{ echo “Time and Date of Tweet: “.$current[‘created_ at’].”
”;

count’].”
”;

echo “Tweet: “. $current[‘text’].”
”; echo “Tweeted by: “. $current[‘user’][‘name’].”
echo “Friends: “. $current[‘user’][‘friends_ count’].”
”;

/>”;

echo “Listed: “. $current[‘user’][‘listed_ echo “Screen name: “. $current[‘user’][‘screen_

count’].”
”;

name’].”
”; echo “Followers: “. $current[‘user’][‘followers_

} ?>


83

Developers Insight Datasets: Array, CSV, les, datasets for research: Iris, Wine and Glass

KNearestNeighbors Classifier in PHP-ML

Figure 13: Portal of PHP-ML for machine learning using PHP

PHP based machine learning on the OpenShift PaaS Machine learning is a powerful science that makes use of soft computing and meta-heuristic approaches for effectual predictive mining even from a huge dataset (https://php-ml. readthedocs.io/en/latest/). It has been traditionally used for fraud detection, market analytics, email spam ltering, malware analysis, ngerprint evaluation, face detection and many other applications. In machine learning, the algorithms are implemented in a way in which better classications and predictions can be made—somewhat similar to the intelligence of the human brain. In traditional implementations, articial neural networks are used with machine learning to solve complex classication problems. A number of libraries and frameworks are available under Free and Open Source Software (FOSS) distribution for machine learning. FOSS libraries that can be integrated for research and development include PHP-ML, Apache Mahout, Shogun, Apache Singa, Apache Spark Mlib, TensorFlow, Oryx2, Accord.NET, Amazon Machine Learning, ScikitLearn, H2O, ConvNetJS, etc. PHP-ML is a powerful machine learning library used for R&D in the domain of machine learning for different applications. It integrates assorted algorithms in the form of classes and methods for high performance computing with the analytics from real-time datasets. PHP-ML has a rich set of algorithms implemented in PHP Scripts, and these can be easily integrated on the real-time cloud of OpenShift by uploading the code and mapping with GIT.

Key features and algorithms in PHP-ML Association rule learning: Apriori Classifcation: KNearestNeighbors, Naive Bayes, SVC, etc Clustering: k-Means, DBSCAN Cross validation: Random split, stratied random split Feature extraction: Token count vectoriser, Tf-idf transformer Metric: Accuracy, confusion matrix, classication report Models management: Persistency Math: Distance, matrix, set, statistic Neural network: Multi-layer perceptron classier Preprocessing: Normalisation, imputation missing values Regression: Least squares, SVR Workow: Pipeline

84

KNearestNeighbors implements the k-nearest neighbours (k-NN) algorithm for solving the classication problems for a specic set of data items. In the following example, inputs with their corresponding targets are specied in terms of classes ‘0’ or ‘1’. If these values are carefully analysed, the corresponding classes can be mapped. In the dataset of [2, 5], [3, 6], [4, 7], the values in each set are in increasing order and there is a difference of +3, which is assigned to the class ‘0’. Similarly, in [4, 2], [5, 3], [7, 5], the values in each set are decreasing and assigned class ‘1’. This data can be trained using k-NN with the implementation of the train() function. $input = [[2, 5], [3, 6], [4, 7], [4, 2], [5, 3], [7, 5]]; $target = [‘0’, ‘0’, ‘0’, ‘1’, ‘1’, ‘1’]; $classifer = new KNearestNeighbors(); $classifer->train($input, $target);

For the prediction of new input data, the predict() function is implemented. As in the following example, predict ([5, 7]) is passed as input, the output will be returned as class ‘0’ because the values in [5, 7] are in increasing order and almost of the same behaviour as class ‘0’. The exact difference of +3 is not mandatory, because machine learning approaches make use of results with a higher degree of approximation, probability and optimisation. $classifer->predict([[10, 6], [1, 3]]); // The function will return [‘1’, ‘0’] depending upon the pattern and behavior of input

Scope of R&D As there are many applications for which classification and predictive mining can be used, the free and open source libraries can be integrated on the real-time clouds of Red Hat OpenShift, IBM Bluemix, Amazon, Google Apps Engine and many others, depending upon the algorithms to be used. The aspects and logs associated with performance, complexity, security and integrity can be analysed with the implementation of algorithms on real-time clouds.

By: Dr Gaurav Kumar The author is the MD of Magma Research and Consultancy, Ambala. He is associated with various academic and research institutes, where he delivers lectures and conducts technical workshops on the latest technologies and tools. You can contact him at [email protected]. Website: www.gauravkumarindia.com.


Insight Developers

A Few Tips for Scaling Up Web Performance Every millisecond counts when it comes to loading Web pages and their responsiveness. It has become critical to optimise the performance of Web applications/pages to retain existing visitors and bring in new customers. If you are eager to explore the world of Web optimisation, then this article is the right place to start.

T

he World Wide Web has evolved into the primary channel to access both information and services in the digital era. Though network speed has increased many times over, it is still very important to follow best practices when designing and developing Web pages to provide optimal user experiences. Visitors of Web pages/applications expect the page to load as quickly as possible, irrespective of the speed of their network or the capability of their device. Along with quick loading, another important parameter is to make Web applications more responsive. If a page doesn’t meet these two criteria, then users generally move out of it and look for better alternatives. So, both from the technical and economical perspectives, it becomes very important to optimise the responsiveness of Web pages. Optimisation cannot be thought of just as an add-on after completing the design of the page. If certain optimisation practices are followed during each stage of Web page development, these will certainly result in a better performance. This article explores some of these best practices to optimise the performance of the Web page/application. Web page optimisation is an active research domain in which there are contributions from so many research groups. An easy-to-use Web resource to start with the optimisation

of Web pages is provided by Yahoo ( https://developer.yahoo. com/performance/rules.html). There are other informative resources, too, such as BrowserDiet (https://browserdiet.com/ en/#html). Various other factors that contribute to Web page optimisation are shown in Figure 1.

Content optimisation When responding to end user requests, the most time is taken up by the downloading of components such as images, scripts, Flash and style sheets. The greater the number of HTTP requests, the more the time required for the page to load and its responsiveness lessens. A critical mechanism to reduce the number of HTTP requests is to reduce the number of components in the Web page. This may be achieved by combining several components. For example, all scripts can be combined, many CSS style sheets can be merged together, etc. Minimising the DNS lookup is another important factor in optimisation. The primary role of Domain Name Systems is the mapping of human readable domain names to IP addresses. DNS lookups generally take somewhere between 20 and 120 milliseconds. Minimising the number of unique host names will reduce the number of DNS 




85

Developers Insight

Content

Images



Optimisations Dimensions

JavaScript

CSS

Server

Mathematical expressions in CSS are evaluated a lot more times than the developer might actually expect. Avoid them as far as possible. If you have to include multiple CSS les, merge them all into one le. This reduces the number of HTTP requests. For example, instead of the following code…

Cookie

Figure 1: Dimensions of Web page optimisations







resolution attempts. Reducing the redirects can increase speed. These redirects are performed with 301 and 302 status codes. With respect to Web 2.0 applications, caching of AJAX (Asynchronous JavaScript And XML) requests is an important step. The number of DOM (Document Object Model) elements should be kept under control.

…use the command given below: 

Opt to use over the @import when using CSS in a page.

Server optimisation 







Using a Content Delivery Network (CDN) can help in optimising the Web page’s performance. Geographical proximity to the user has a positive impact on the time required to fetch content. A cache-control header can help. If the content is static, then the expiry should be set as Never Expire. For dynamic content, the time up to when the component is valid should be set. This will minimise HTTP requests. Compressing the components is another great step in optimisation. This can be achieved with ‘Gzip’. Experts estimate that the compression minimises the time required for responses by 70 per cent. With respect to AJAX applications, the GET method is preferable. So, along with XMLHttpRequest , as far as possible use the GET method.

Cookies Cookies are one of the most used mechanisms by Web developers to store tiny pieces of information. With respect to cookies, the following factors should be considered: Size of the cookies should be kept minimal. Cookies should be set at the appropriate level in the domain hierarchy. This is done to reduce the impact on sub-domains. Don’t forget to set a proper expiry date for the cookie.

JavaScript JavaScript has become the de-facto client-side scripting language. So the way in which JavaScript components are built does have a signicant impact on the performance of Web pages. If possible, move the script to the bottom of the page. This cannot be done always (for example, if your page’s critical contents are rendered through the document. write() function). Using external JavaScript and style sheet les will enable better caching. So, it would be better in many scenarios to put CSS and JavaScript through the external mode. Minifying and Obfuscation are two effective mechanisms to improve the performance by tweaking the code. One survey indicates that obfuscation can achieve a 25 per cent reduction in size. Crowding of events needs to be avoided. Delegating events properly improves the performance of the page. The usage of async (asynchronous) must be encouraged, as shown below: 









 



Style sheets Professionally designed style sheets make Web pages look elegant. The following factors must be considered in handling style sheets: It is better to keep the style sheets in the HEAD section of the Web pages. This is done to permit the pages to render incrementally. Care should be taken to use expressions in CSS.

<script async src=”example.js”>

If you don’t use the aysnc keyword then the page has to wait till the example.js is fully downloaded. The aysnc keyword makes page parsing happen even before the downloading of the script is completed. Once the script is downloaded, it is activated. However, when using multiple async, the order of execution becomes a concern.



Optimising images



Images are an integral part of most Web pages. Hence, the way images are handled denes the performance of the application. The following factors should be considered:

86


Insight Developers 



Scaling down of images using HTML tags should be avoided. There is no point in using a bigger image and resizing it using the width and height attributes of the tag. When using Data URI, the contents can be given in inline mode. This can be done for smaller sized images. So, instead of the following command…

.icon-test {

Diagnostic Tools Lighthouse

Tool

AAAAASUVORK5CYII%3D’); }

Images generally contain data that are not required in Web usage. For example, the EXIF metadata can be stripped before uploading to the server. There are many tools to help you optimise images, such as TinyPNG, Compressor.io, etc. There are command line based tools also, such as jpegtran, imgopt, etc.

   

SpeedCurve

Description

Apache JMeter

This load testing tool is popular in the Java community.

Locust

This load testing tool can be used to specify user behaviour with Python. The capability to handle millions of simultaneous user requests can be tested with this tool.

Wrk

This is an HTTP benchmarking tool.

HTTPerf

Different types of HTTP workloads shall be generated and tested. There are various ports available for HTTPerf: HTTPerf.rb: Ruby interface HTTPerf.py: Python Port HTTPerf.js: JavaScript Port Gohttperf: Go port

Performance analysis tools



Table 1

background-image: url(‘data:image/png;bas

There are many tools available to analyse the performance of Web pages. Some of these tools are illustrated in Figure 2. There are component-specic tools, too. For example, for benchmarking JavaScript, the following tools may be used: JSPerf Benchmark.js JSlitmus Matcha Memory-stats.js For PHP, tools such as PHPench and php-bench could be harnessed.

WebPageTest

Figure 2: Performance analysis tools

MVEUAAACnej3aAAAAAXRSTlMAQObYZgAAAApJRE FUCNdjYAAAAAIAAeIhvDM



PageSpeed

DareBoost

e64,iVBORw0KGgoAAAANSUhEUgAAAAEAAAABAQMAAAAl21bKAAAAA1B



YSlow

Yellow Lab Tools

background-image: url(‘test.png’); }

…use the code given below: .icon-test {

Confess

• • • •

Benchmarking Web servers Benchmarking of Web servers is an important mechanism in Web page/application optimisation. Table 1 provides a sample list of tools available for benchmarking Web servers. The Web optimisation domain is really huge. This article has just provided a few start-up pointers, using which interested developers can proceed further in understanding the advanced technicalities of the topic.

Minifiers As stated earlier, minifying is one of the optimisation techniques, for which there are many tools. For HTML, the following Miniers could be tried out: HTMLCompressor HTMLMinier HTML_press Minimize Some of the tools used for Minifying JavaScript and CSS are listed below: Uglifyjs2 CSSmin.js Clean-css JShrink JSCompress YUI Compressor    

    

References [1] Yahoo Best Practices: https://developer.yahoo.com/ performance/rules.html [2] Browser Diet: https://browserdiet.com/en/#html [3] https://github.com/davidsonfellipe/awesomewpo#analyzers [4] https://github.com/zenorocha/browser-diet/wiki/ Tools#optimize-your-images

By: Dr K.S. Kuppusamy The author is assistant professor of computer science, School of Engineering and Technology at Pondicherry Central University. He has vast experience in teaching and research (in academia and industry). He can be reached via mail at [email protected].




87

For U & Me

Success Story

Open Source Enables PushEngage to Serve 20 Million Push Notifications Each Day! Bengaluru-based PushEngage has deployed open source technologies such as Apache, Bootstrap, MongoDB, MySQL, Node.js and Nginx to handle its push notification service. More than half of its clients are based outside the country.

T

he digital marketing world has been ruled by the electronic direct mailer (eDM) for quite some time. But the eDM is now making way for push notications. Be it an e-commerce site or an online news publication, Web masters are deploying push notications to grow their trafc as well as enhance their brand. But what does a push notication provider deploy to serve a bulk of notications? Well, it is usually an open source solution! Bengaluru-based PushEngage is among the few early adopters of push notications. The company had built an in-house product to test the success rate of notications circulated over-the-air at the time when Google offered the same support to Chrome in April 2015. The initial results were strong enough to commercialise the product. “We saw robust results even at the

88

early stage, which is when we started considering building PushEngage and went on to create an automated marketing platform for browser-based push notications, available to all,” recalls Ravi Trivedi, founder and CEO, PushEngage. With a small team of just 10 employees, Trivedi’s PushEngage handles marketing automation through notications for more than 6,000 clients around the world. The total client base sends over 20 million notications on a daily basis. All of that comes from 40 servers that run in the cloud, and use a mix of proprietary and open source solutions at the back-end. A 10-member team handles more than 6,000 global clients to serve over 20 million push notifications on a daily basis!


The prime reason behind the mountainous growth of PushEngage is the ease of its deployment on any website. Local search site AskLaila, which receives over a million monthly unique visits, claims that notications through PushEngage can be deployed in as early as ten minutes. The service has also helped the company retain its users. “With PushEngage notications, we have been able to reach out to users who are not active on the site and provide them with helpful offers or information,” says Nitin Agrawal, director of engineering, Asklaila.com.

Bringing community offerings to the mainstream Trivedi tells Open Source For You that while his company had initially chosen components that helped to scale better, along with a faster development time,

Success Story the team has now planned to look for community-based alternatives once the product and its technology stack mature. “We began with proprietary solutions so that someone could handle the complexity and scale in the beginning. For instance, using AWS (Amazon Web Services) Kinesis, instead of Apache Kafka, provides multiple shards and easy scalability,” the founder mentions. Being a great fan of open source technologies since the time he did his master’s in computer science at the Indian Institute of Science in Bengaluru, Trivedi wants to rely entirely on open source. PushEngage already uses the Nginx server to scale connections for sending push notications to Web browsers, and there are plans to switch to Apache Kafka from the proprietary Kinesis solution — to use a server architecture solely based on open source technologies.

The process of enabling automation PushEngage uses open APIs provided by Web browsers such as Chrome, Firefox and Safari that adhere to the W3C-Push standards. In addition to leveraging the available APIs, the company uses its internally built libraries that have Node.js as a programming language and MongoDB as a data store. “The open source technologies behind PushEngage help us become more competitive as these solutions are tested thoroughly for scale, security and bugs. The deployment of open source also reduces our maintenance tasks and offers us the freedom to move across any cloud provider without a vendor lock-in,” Trivedi says.

Challenges of mass adoption Even though open source has made PushEngage capable of delivering push notications on a massive scale, catering to thousands of clients was not easy for the company in the early stages. “Push notications send and receive messages in bursts, and hence the peak capacities are very high but of a short duration. Building a scalable solution around these issues requires well thought out architecture and solid engineering,” explains Trivedi.

For U & Me

The team at PushEngage that handles millions of notifications each day

The rst version of the automated model was a minimally viable product. However, subsequent versions enabled the engineers to incorporate a highly scalable and efcient combination that helps to send and receive messages on a large scale. Apart from building just a scalable solution, the PushEngage team was required to set up a datastore that could generate queries on several attributes in real-time and push messages at a fast pace. There was also a need for datastores that had high read characteristics.

re-invent the wheel,” says Trivedi. PushEngage is also committed to giving back to the community and developers worldwide. The company has already provided a push notication API free to developers. This allows any developer to integrate all the rich features of customer segmentation, automation, scheduling and triggering notications, as well as geotargeting.

Open source adoption cuts costs According to Trivedi, deploying open source helped his company to reduce the cost of the project by 25 per cent. “Open source adoption enabled PushEngage to reduce not just initial operational costs but also our ongoing cost of product development,” he notes.

Future-proof PushEngage enables automation on the Web

PushEngage moved to a microservicepowered, asynchronous message-based architecture to overcome the primary challenges. “The deployment of appropriate architecture provided us with the desired burst-mode scalability as well as fast enough results for our customers,” says Trivedi.

The role of the community To build a secure and advanced logging system, as well as testing frameworks for automating notications, PushEngage took help from the open source community, accessing libraries on GitHub and SourceForge. “Online listings of open source developments are quite handy in nding good libraries, so we don’t have to

Moreover, open source is making the automation process through PushEngage future-proof. “We have a microservicebased architecture built for horizontal scaling. We also use queue-based message passing, as well as datastores that are strong in both read and write characteristics and have a scalable database service. All this ensures our scaling needs are well met,” says Trivedi. The company also uses relational and non-relational databases, depending on the need of the architecture. All this makes the push notication technology capable of expanding, with the addition of new components and features in the future.

By: Jagmeet Singh The author is an assistant editor at EFY .


89

Developers Insight

Regular Expressions in Programming Languages: The Story of C++ In this issue of OSFY, we present the third article on regular expressions in programming languages. The earlier articles covered the use of regular expressions in general, in Python and then in Perl. Read on to discover the intricacies of regular expressions in C++.

I

nterpreted languages often have weakly typed variables which don’t require prior declaration. The additional benet of weakly typed variables is that they can be used to hold different types of data. For example, the same variable can hold an integer, a character, or a string. Due to these qualities, scripts written in such languages are often very compact. But this is not the case with compiled languages, for which you need a lot of initialisation; and with strongly typed variables, the code is often longer. Even if the regular expression syntax for interpreted and compiled languages is the same, how they are used in real programs is different. So, I believe it is time to discuss regular expressions in compiled languages. In this article, I will discuss the regular expression syntax of C++.

Standards of C++ People often fail to notice the fact that programming languages like C and C++ have different standards. This is quite unlike languages like Perl and Python, for which the use of regular expressions is highly warranted due to the very nature of these programming languages (they are scripting languages widely used for text processing and Web application development). For a language like C++, heavily used for highperformance computing applications, system programming, embedded system development, etc, many felt that the inclusion of regular expressions was unnecessary. Many of the initial standards of C++ didn’t have a natural way for handling regular expressions. I will briey discuss the different standards of C++ and which among them has support for regular expressions. C++ was invented by Bjarne Stroustrup in 1979 and was initially known as ‘C with Classes’ and later renamed C++ in

90

1983. A book titled ‘The C++ Programming Language’, rst published in 1985 by Stroustrup himself, and its subsequent editions became the de facto standard for C++ until 1998, when the language was standardised by the International Standards Organization (ISO) and the International Electrotechnical Commission (IEC) as ISO/IEC 14882:1998, informally called C++98. The next three standards of C++ are informally called C++03, C++11 and C++14. Hopefully, by the time this article gets published, the latest standard of C++, informally called C++17, would have been released and will have some major changes to C++. After this, the next big changes in C++ will take place with a newer standard, informally known as C++20, which is set to be released in 2020. The rst three standards of C++, namely the de facto standard of C++, C++98 and C++03, do not have any inbuilt mechanism for handling regular expressions. Things changed with C++11 when native support for regular expressions was added with the help of a new header le called . In fact, the support for regular expressions was one of the most important changes brought in by this standard. C++14 also has provision for native support of regular expressions, and it is highly unlikely that C++17 or any future standards of C++ will quash the support for handling regular expressions. One problem we might face in this regard is that the academic community in India mostly revolves around the C++98 standard, which doesn’t support regular expressions. But this is just a personal opinion and I don’t have any documented evidence to prove my statement.

The C++11 standard Unlike C++03 and C++14, for which the changes were minimal, C++11 was a major revision of C++. GCC 5 fully


Insight Developers supports the features of C++11and C++14. The latter has become the default standard for GCC 6. There were many changes made to the core language by the standard C++11. The inclusion of a new 64-bit integer data type, called long long int, is a change made in C++ by the C++11 standard. Earlier, C++ only had 32-bit integers called long int. External templates were also added to C++ by this standard. Many more changes were made to the core of the C++ language by the C++11 standard, but the changes were not limited to the core alone — the C++ standard library was also enhanced by the C++11 standard. Changes were made to the C++ standard library in such a way that multiple threads could be created very easily. New methods for generating pseudo-random numbers were also provided by the C++11 standard. A uniform method for computing the return type of function objects was also included by the C++11 standard. Though a lot of changes have been made to the standard library in C++11, the one that concerns us the most is the inclusion of a new header le called .

programs. This and all the other C++ programs and text les used in this article can be downloaded from opensourceforu. com/article_source_code/September17C++.zip. #include #include using namespace std; int main( ) { char str[ ] = “Open Source For You”; regex pat(“Source”); if( regex_search(str,pat) ) { cout << “Match Found\n”; }

else { cout<<”No Match Found\n”; }

Regular expressions in C++11 In C++, support for regular expressions is achieved by making changes to the standard library of C++. The header le called is added to the C++ standard library to support regular expressions. The header le is also available in C++14 and, hence, what we learn for C++11 also applies to C++14. There are some additions to the header le in C++14, which will be discussed later in this article. There are three functions provided by the header le . These are regex_match( ), regex_search( ) and regex_replace( ). The function regex_match( ) returns a match only if the match is found at the beginning of a string, whereas regex_search( ) searches the entire string for a match. The function regex_replace( ) not only nds a match, but it replaces the matched string with a replacement string. All these functions use a regular expression to denote the string to be matched. Other than these three functions, the header le also denes a number of classes like regex, wregex, etc, and a few iterator types like regex_iterator and regex_token_ iterator. But to simplify and shorten our discussion, I will only cover the class regex and the three functions, regex_ search( ), regex_match( ) and regex_replace( ). I believe it is impossible to discuss all the features of the header le in a short article like this, but the topics I will cover are a good starting point for any serious C++ programmer to catch up with professional users of regular expressions. Now let us see how regular expressions are used in C ++ with the help of a small C++ program.

A simple C++ program using regular expressions The code below shows a C++ program called regex1.cc. I am sure you are all familiar with the .cc extension of C++

return 0; }

I’m assuming that the syntax of C is quite well known to readers, who will understand the simple C++ programs we discuss in this article, so no further skills are required. Now let us study and analyse the program. The rst two lines #include and #include include the two header les and . The next line of code using namespace std; adds the std namespace to the program so that cout, cin, etc, can be used without the help of the scope resolution operator (::). The line int main( ) declares the only function in this program, the main( ) function. This is one problem we face when programming languages like C++ or Java are used. You need to write a lot of code to set up the environment and get things moving. This is one reason why you should stick with languages like Perl or Python rather than C++ or Java if your whole aim is to process a text le. But if you are writing system software and want to analyse a system log le, then using regular expressions in C++ is a very good idea. The next line of code char str[ ] = “Open Source For You”; initialises a character array called str[ ] with a string in which we will search for a pattern. In this particular case, the character array is initialised with the string Open Source For You. If you wish to replace the line of code char str[ ] = Open Source For You with string str = “Open Source For You”; and thereby use an object str of string class of C++ instead of a character array, the program will still work equally well. Remember that the string class of C++ is just an instance of the template class basic_string. This modied program called string.cc is also available for downloading. On execution with the commands g++ string.cc and . /a.out, the program string.cc will also behave www.OpenSourceForU.com | OPEN SOURCE FOR YOU | SEPTEMBER 2017 |

91

Developers Insight this case, the word Source appears as the second word in the string Open Source For You, and hence no match is found by the function regex_match( ). Figure 1 shows the output of the programs regex1.cc and regex2.cc.

Pattern replacement in C++ Figure 1: Output of regex_search( ) and regex_match( )

just like the program regex1.cc. Since I am expecting a mixed group of readers with expertise in different programming languages, I tried to make the C++ programs look as much as possible like C programs, in the belief that C is the language of academia and everybody has had a stint with it as a student. I could have even used the printf( ) and scanf( ) functions instead of cout and cin. But a line should be drawn somewhere, and this is where I have stopped making C++ programs look like C. The next line of code regex pat(“Source”); is very important. It is responsible for setting up the regular expression pattern that should be searched in the string Open Source For You. Here the pattern searched is the word ‘Source’ which is stored in an object called pat of the class regex. The next few lines of code contain an if-else statement. The line of code if( regex_search(str,pat) ) uses the function regex_ search( ) provided by the header le to search for the pattern stored in the object pat of the class regex in the string stored inside the character array str[ ]. If a match is found, the line of code cout << “Match Found\n”; is executed and prints the message Match Found. If a match is not found, the else part of the code cout << “No Match Found\n”; is executed and prints the message No Match Found. This program can be compiled with the command g++ regex1.cc, where g++ is the C++ compiler provided by GCC (GNU Compiler Collection). This will produce an executable called a.out . This is then executed with the command ./a.out . On execution, the program prints the message Match Found on the screen because the function regex_search( ) searches the entire string to nd a match. Since the word Source is present in the string Open Source For You, a match is found. Now, it is time for us to revisit the difference between the functions regex_search( ) and regex_match( ). To do this, the line of code if( regex_search(str,pat) ) in the program regex1. cc is replaced with the line if( regex_match(str,pat) ). This modied code is available as a program named regex2.cc, which can be compiled with the command g++ regex2.cc, which will produce an executable called a.out . This is then executed with the command ./a.out . Now the output printed on the screen is No Match Found. Why? As mentioned earlier, this is due to a difference between the functions regex_search( ) and regex_match( ). The function regex_search( ) searches the entire string for a match and the function regex_match( ) returns true only if the regular expression pattern is present at the beginning of a string. In

92

Let’s now study the working of the function regex_replace(). Consider the program regex3.cc which uses the function regex_replace( ). This function will search for a match and if it nds one, the function will replace the matched string with a replacement string. #include #include #include using namespace std; int main( ) { char str[ ] = “Open Source Software is Good”; regex pat(“Open Source”); char rep[ ] = “Free”; cout <
Except for the line of code cout < #include #include using namespace std;


Developers Insight int main( ) {

return 0; }

char str[ ] = “Open Source Software is Good”; regex pat(“Good”); char rep[ ] = “Excellent”; cout <
On execution with the commands g++ regex4.cc and ./a.out, the program regex4.cc prints the message Open Source Software is Excellent . This clearly tells us that the function regex_replace() behaves like the function regex_search( ) whereby the whole string is searched for a possible match for the given regular expression, unlike the function regex_match( ) which looks for a match at the very beginning of a string. Figure 2 shows the output of the two programs, regex3.cc and regex4.cc.

File processing in C++ with regular expressions

When the program regex5.cc is executed with the commands g++ regex5.cc and ./a.out, the message printed on the screen is UNIX IS AN OPERATING SYSTEM . So, a case-sensitive pattern match is carried out here. The next question is: How do we carry out a case-insensitive pattern match? For this purpose, we use a regex constant called icase. When the line of code regex pat(“UNIX”); is replaced with the line regex pat(“UNIX”, regex_constants::icase); a case insensitive match is carried out, and this results in a match for three lines in the text le fle1.txt . Figure 3 shows the results of the case-sensitive and case-insensitive matches. There are many other regex constants dened in the namespace regex_constants. Some of them are nosubs, optimize, collate, etc. Use of these regex constants will add more power to your regular expressions. It is a good idea to learn more about them as you gain more information about C++ regular expressions.

The next question that needs to be answered is: How do we process data inside a text le with a regular expression? To test the working of such a program, a text le called fle1.txt is used, which is the same one used in the previous articles in this series on regular expressions. Figure 2: The function regex_replace( ) in C++ unix is an operting system Unix is an Operating System UNIX IS AN OPERATING SYSTEM Linux is also an Operating System

Now let us consider the following C++ program called regex5.cc that reads and processes the text le fle1.txt line by line, to print all the lines that contain the word ‘UNIX’.

Figure 3: Case-sensitive and case-insensitive matches

#include

Regular expressions in C++14 and C++17

#include

It is now time for us to discuss regular expressions in C++14. Luckily, except for a few minor additions, the header le of C++11 has remained largely unchanged even after the introduction of the later standard C++14. For example, the denitions of the functions regex_match( ) and regex_search() are slightly modied in C++14 so that additional processing with these functions is possible. But these changes only add more power to existing functions and do not affect their basic working. And nally, what are the changes that will be brought on by C++17? Hopefully, nothing major. So far, there have been no rumours about whether there will be a major revision to the header le . Therefore, whatever we have learnt from this article can be used for a long time.

#include #include using namespace std; int main( ) { ifstream le(“le1.txt”); string str; regex pat(“UNIX”); while (getline(le, str)) { if( regex_search(str,pat) )

Regular expression style of C++

{ cout << str <<”\n”; } }

94

Unlike the previous two articles in this series, in this article I have started by explaining C++ code snippets using regular expressions directly, without providing details regarding the


Insight Developers kind of regular expression syntax being used in C++. Sometimes it is better to attack the problem directly than beat around the bush. But even then, it is absolutely essential to know the regular expression syntax used with C++. Otherwise, this article may just be a set of ‘Do-It-Yourself’ instructions. C++11 regular expressions support multiple regular expression styles like ECMAScript syntax, AWK script syntax, Grep syntax, etc. ECMAScript is a scripting language and JavaScript is the most well-known implementation of ECMAScript. The syntax used by ECMAScript is not much different from the other regular expression avours. There are some minor differences, though. For example, the notation \d used in Perl style regular expressions to denote decimal digits is absent in ECMAScript style regular expressions. Instead, a notation like [[:digit:]] is used there. I am not going to point out any other such difference but just keep in mind that C++11 supports multiple regular expression styles and some of the styles differ from the others, slightly.

A practical regular expression for C++ Now let us discuss a practical regular expression with which we can nd out some real data rather than nding out ‘strings starting with abc and ending with xyz’. Our aim is to identify those lines that contain only numbers. Consider the text le fle2.txt with the following data to test our regular expressions:

while (getline(le, str)) { if( regex_search(str,pat) ) { cout << str <<”\n”; } } return 0; }

On execution with the following commands, g++ regex7.cc and ./a.out the program prints those lines containing numbers alone. Figure 4 shows the output of the program regex7.cc. Except for the line of code regex pat(“^[[:digit:]]+$”); which denes the regular expression pattern to be searched, there is no difference between the working of the programs regex5.cc and regex7.cc. The caret symbol ^ is used to denote that the match should happen at the very beginning and the dollar symbol $ is used to denote that the match should occur at the end. In the middle, there is the regular expression [[:digit:]]+ which implies one or more occurrences of decimal digits, the same as [0-9]+. So, the given regular expression nds a match only if the line of text contains only decimal digits and nothing more. Due to this reason, lines of text like AA111, 22.22, a1234z, etc, are not selected.

abcxyz a1234z 111222 123456 aaaaaaa

Figure 4: Regular expressions for numbers

zzzzzzz AA111 111 2222 33333 22.22 BBBB

Now consider the program regex7.cc with the following code: #include #include #include #include

Now it is time for us to wind up the discussion. Like the previous two articles in this series, I have covered the use of regular expressions in a particular programming language as well as some other aspects of the programming language that will affect the usage of regular expressions. In this article, the lengthy discussion about the standards of C++ was absolutely essential, without which you might blindly apply regular expressions on all standards of C++ without considering the subtle differences between them. The topics on regular expressions discussed in this article may not be comprehensive but they provide an adequate basis for any good C++ programmer to build up from. In the next part of this series we will discuss the use of regular expressions in yet another programming language, maybe one that is much used on the Internet and the World Wide Web.

using namespace std;

By: Deepu Benson int main() { ifstream le(“le2.txt”); string str; regex pat(“^[[:digit:]]+$”);

The author has nearly 16 years of programming experience. He is a free software enthusiast and his area of interest is theoretical computer science. The author maintains a technical blog at www.computingforbeginners.blogspot.in and can be reached at [email protected].


95

For U & Me

Insight

Eight Top-of-the-Line Open Source Game Development Tools The open source game development tools presented in this article give developers numerous options to explore and choose from, as per their requirements.

O

pen source game development is generally looked upon as a tech enthusiast’s hobby. Rapid advancements in technology, combined with the various innovations being launched every day, have put tech experts and gamers in a win-win situation. Open source provides interoperability, high quality and good security in game devlopment. Little wonder then that open source platforms are already being used for quite a few successful and complex games. The following points highlight some of the advantages of the open source gaming platforms. Better quality and more customised software: With the source code being available on open source gaming platforms, professional developers can customise features and add varied plugins as per their own requirements, which is benecial for game development companies. Say good bye to licensing: With completely open source platforms, there is no requirement for any sort of licensing. So apart from no licence costs, other issues like tracking and monitoring are also avoided. Lower cost of hardware: Open source gaming platforms in Linux involve lower hardware costs compared to Windows. With the advantages of easy portability and high compression, Linux requires low hardware congurations. So, game development costs are lower and even legacy hardware systems can be used for game development. Let’s take a look at the top open source gaming development platforms, which give developers numerous options to explore and choose from, as per their requirements. 





96

GDevelop GDevelop is an open source, cross-platform game creator platform designed for novices. There is no requirement for any sort of programming skills. GDevelop is a great platform to develop all sorts of 2D and 3D games. It consists of several editors on which games can be created. The list is as follows. Project manager: This displays the open games in the editor, allowing developers to set and organise the scenes. Users can select the scene to be edited and modify the parameters like the title, background colour, text, etc. It also gives access to the image bank editor of the games, and allows the user to select the extensions to be utilised by the game. Image bank editor: This allows the user to manage all sorts of images via objects. It supports transparency integrated in the image. Scene editor: This allows users to organise the scene at the start, positioning the object in the scene. Object editor: This allows the creation of objects to be displayed on the stage, like text and 3D box objects. It also has the ‘Particle Transmitter’ object, which allows developers to use particles in the game with ease. Layer editor: This allows users to manage the interface that remains motionless, while allowing the camera of the rest of the game to move or zoom. Event editor: This allows users to animate the scene, depending on the conditions and actions that will be performed on the objects of the scene. The events are compiled by GDevelop in machine code — the mechanism is simple and similar to writing C++ code. 












For U & Me

Insight

Features 







It comprises various objects which can be used readily — text objects, 3D boxes, own customised shapes via Shape Painter, the particle engine, dynamic lights and shadows, custom collision masks, etc. Adds behaviours to the objects through the physics engine, pathnding, top-down movement, platformer engine, draggable objects and the automation of tasks. Offers advanced design features and interfaces through the scene editor, multiple layers, the debugger and performance prolers. Other features include HTML 5 support, sound and music effects, and integration with the joystick and keyboard.

Features 







Latest version: 4.0.94

Ofcial website: http://compilgames.net/



Nice and clean interface: Has a visual editor, a dynamic scene system, a user friendly content creation interface, a visual shader editing tool and live editing on mobile devices. Efcient in 2D game design because of a dedicated 2D engine, custom 2D physics engine, and a exible kinematic controller. High-end 3D game development by importing animation models from 3DS Max, Maya and Blender; has skeleton deforms and blend shapes, lighting and shadow mapping, HDR rendering, anti-aliasing, etc. Flexible animation engine for games, enabled by the visual animation editor, frame-based or cut-out animation, custom transition curves and tweens, and animation tree support. Other features include a Python-like scripting language, a powerful debugger and an easy C++ API.


Ofcial website: https://godotengine.org

Figure 1: GDevelop user interface

Godot Engine The Godot Engine is a highly powerful cross-platform game development engine that supports 2D and 3D game development from a unied interface. The platform supports Windows, OS X, Linux and BSD for developing games for the PC, console and even mobile-cum-Web platforms. It is integrated with a wide variety of tools, providing developers with tons of options, and avoiding the need for even a single third party tool. The engine is built on the concept of a tree of nested scenes. The games created with Godot are either written in C++ or a customised scripting language called GDScript, which is a high level, dynamically typed language with many similarities to Python. GDScript is greatly customised and optimised for the Godot engine. Godot has a power text editor, which provides developers various features like auto indentation, highlighting syntax and even code completion. It also has a debugger to provide breakpoints and program stepping. Godot makes use of the OpenGL ES 2.0 graphics engine, which has many features like transparency, normal mapping, dynamic shadows using shadow maps, and various postprocessing effects like FXAA, bloom, DOF, HDR, gamma correction and fog.

98

Figure 2: Godot Engine interface

Cocos2d-x Cocos2d-x is an open source game development platform available under the MIT License. It allows developers to build games, apps and various interactive programs. It enables developers to make use of C++, Lua and JavaScript for crossplatform deployment on iOS, Android, Windows Phone, OS X, Windows and Linux devices. The cocos2d-x renderer engine is highly optimised for 2D graphics with OpenGL support. It is packed with tons of features like skeletal animation, sprite sheet animation, coordinate systems, visual effects, textures, tile maps, multiresolution devices, etc. It is maintained by developers at Chukong Technologies, which is also developing Cocostudio, a WYSIWYG editor. Features 




Animation: It provides numerous animation options that work on sprites using a set of actions and timers. It supports animation of particle effects, image ltering effects through shaders, etc. Easy GUI: It includes an easy GUI interface with text

www.IndiaElectronics Week .com

DRIVING TECHNOLOGY INNOVATION & INVESTMENTS

Colocated shows

Profit from IoT India’s #1 IoT Show. At Electronics For You, we strongly believe that India has the potential to become a super power in the IoT space in the upcoming years. All that's needed are platforms for different stake-holders of the eco-system to come together. We’ve been building one such platform: IoTshow.in--an event for the creators, the enablers and customers of IoT. In Feb 2018, the 3rd edition of IoTshow.in will bring together a B2B expo, technical and business conferences, Start-up Zone, demo sessions of innovative products, and more. Who Should Attend? •

Creators of IoT Solutions: OEMs,

Design Houses, CEOs, CTOs, Design Engineers, Software Developers, IT Managers, etc • Enablers of IoT Solutions: System Integrators, solution providers, distributors, resellers, etc • Business Customers: Enterprise, SMEs, Government, Defence, Academia, etc Why Should You Attend? • Get updates on latest technology trends defining the IoT landscape • Get a first-hand glimpse of products & solutions that enable development of better IoT solutions • Connect with leading IoT brands seeking channel partners and system integrators • Connect with leading suppliers/service-providers of electronics, IT and telecom services who can help you produce IoT solutions better and faster • Network with the who’s who of the IoT world and build connects with industry peers • Find out IoT solutions that can help you reduce costs or increase revenues • Get updates on latest business trends shaping demand and supply for IoT solutions

India’s Electronics Manufacturing Show Is there a show in India that showcases the latest in electronics manufacturing such as Rapid Prototyping, Rapid Production and Table Top Manufacturing? Yes, there is now - EFY Expo 2018. EFY Expo 2018, with its focus on the said areas and being co-located at India Electronics Week, has emerged as India's leading show on latest manufacturing technologies and electronics components Who Should Attend? • Manufacturers: CEOs, MDs etc—who are manufacturing electronics & technology products • Purchase Decision Makers: CEOs, Purchase Managers, Production Managers, etc involved in electronics manufacturing • Technology decision makers: Design engineers, R&D heads etc. – involved in electronics manufacturing • Channel Partners: Importers, distributors, resellers of electronics components, tools & equipment • Investors: Start-ups, Entrepreneurs, Investment Consultants interested in electronics manufacturing Why Should You Attend? • Get updates on latest technology trends in rapid prototyping & production and table top manufacturing • Get connected with new suppliers from across India to improve your supply-chain • Connect with OEMs, principles, brands seeking channel partners and distributors • Connect with foreign suppliers and principles to represent them in India • Explore new business ideas and investment opportunities in this sector

Showcasing Technology that Powers the Light Our belief is that the LED Bulb itself is a culmination of advancement in technology. And, such a product category and its associated industry cannot grow without focusing on latest technologies. But, while there are some good B2B shows for LED Lighting in India, none has a focus on “the technology that powers the light”. Thus, the need for LEDasia.in. Who Should Attend? •

•

• •

•

Tech Decision Makers: CEOs, CTOs, R&D,

Design Engineers, etc—who are developing latest LED-based products Purchase Decision Makers: CEOs, Purchase Managers, Production Managers, etc from manufacturers that use LEDs Channel Partners: Importers, distributors, resellers of LEDs & LED Lighting products Investors: Start-ups, Entrepreneurs, Investment Consultants interested in this sector Enablers: System Integrators, Lighting consultants, etc interested in Smarter Lighting Solutions (thanks to co-located IOTshow.in)

Why Should You Attend? • Get updates on latest technology trends defining LED & LED Lighting sector • Get a first-hand glimpse of latest components, equipment and tools to help produce better lighting products • Get connected with new suppliers from across India to improve your supply-chain • Connect with OEMs, principles, lighting brands seeking channel partners and system integrators • Connect with foreign suppliers and principles to represent them in India • Explore new business ideas and investment opportunities ideas in LED & Lighting sector • Get a first-hand view of “IOT + Lighting” solutions that make lighting smarter


Colocated shows

Reasons Why You Should NOT Attend IEW 2018

India’s Mega Tech Conference EFY Conferences (EFYCON) started as a tiny 900-footfall community conference in 2012, going by the name of Electronics Rocks. Within four years it grew into “India’s largest, most exciting engineering conference,” and was awarded as the most important IoT global event in 2016 by Postscapes. In 2017, 11 independent conferences on IOT, Artificial Intelligence, Cyber Security, Data Analytics, Cloud Technologies, LED Lighting, SMT Manufacturing, PCB Manufacturing, etc were held together in 3 days, as part of EFY Conferences. Key Themes of Conferences & Workshops in 2018 • Profit from IoT: How can suppliers make money and customers save money using IoT • IT & Telecom Tech Trends Enabling IoT Development • Electronics Tech Trends Enabling IoT Development • Artificial Intelligence and IoT • Cyber-security and IoT • Latest Trends in Test & Measurement Equipment • What's New in Desktop Manufacturing • The Latest in Rapid Prototyping & Production Equipment Who Should Attend? • Investors & Entrepreneurs in Tech • Technical Decision Makers & Influencers • R&D Professionals • Design Engineers • IOT Solution Developers • System Integrators • IT Managers

SPECIAL PACKAGES FOR • Academicians • Defence Personnel • Bulk/Group Bookings

We spoke to few members of the tech community to understand why they did not attend past editions of India Electronics Week (IEW). Our aim was to identify the most common reasons and share them with you, so that if you too had similar reasons, you may choose not to attend IEW 2018. This is what they shared… #1. Technologies like IOT, AI, Embedded Systems Have No Future Frankly, I have NO interest in new technologies like Internet of Things (IOT), Artificial Intelligence, etc. I don't think these will ever take off, or become critical enough to affect my organization or my career. #2. I See No Point in Attending Tech Events What's the point in investing energy and resources to attend tech events? I would rather wait and watch—-let others take the lead. Why take the initiative to understand new technologies, their impact and business models—beats me.

of the speakers are not vendors? Where most talks will not be by people trying to sell their products? How boring! I can't imagine why anyone would want to attend such an event. I love sales talks, and I am sure everybody else does too. So IEW is a big 'no-no' for me. #7. I Don't Need Hands-on Knowledge I don't see any value in tech workshops being organised at IEW. Why would anyone want hands-on knowledge? Isn't browsing the Net and watching YouTube videos a better alternative? #8. I Love My Office Why do people leave the comfort of their office, and weave through that terrible traffic to attend a technical event? They must be crazy. What’s the big deal in listening to experts or networking with peers? I'd rather enjoy the coffee and the cool comfort of my office, and learn everything by browsing the Net!

#3. My Boss Does Not Like Me My boss is not fond of me and doesn't really want me to grow professionally. And when she came to know that IEW 2018 is an event that can help me advance my career, she cancelled my application to attend it. Thankfully, she is attending the event! Look forward to a holiday at work.

#9. I Prefer Foreign Events While IEW's IOTshow.in was voted as World's #1 IOT event on Postscapes.com, I don't see much value in attending an event in India—and that, too, put together by an Indian organizer. Naah! I would rather attend one in Europe or an event organized by foreigners.

#4. I Hate Innovators Oh my! Indian start-ups are planning to give LIVE demonstrations at IEW 2018. I find that hard to believe. Worse, if my boss sees these, he will expect me to create innovative stuff too. I better find a way to keep him from attending.

Hope we've managed to convince you to NOT to attend IEW 2018. Frankly, we too, have NO clue why 10,000-plus techies attended IEW in March 2017. Perhaps there's something about the event that we've not figured out yet. But, if we haven't been able to dissuade you from attending IEW 2018, then you may register at http://register.efy.in.

#5. I Am Way Too BUSY I am just too busy with my ongoing projects. They just don't seem to be getting over. Once I catch up, I'll invest some time in enhancing my knowledge and skills, and figure out how to meet my deadlines. #6. I Like Vendor Events Can you imagine an event where most

Conference Pass Pricing

1 Day Pass

INR 1999 PRO Pass

INR 7999

Special Privileges & Packages For Defence & Defence Electronics Personnel Academicians

Group & Bulk Bookings


Colocated shows

The Themes • Profit from IOT

• Rapid Prototyping & Production

• Table Top Manufacturing

• LEDs & LED Lighting

The Co-located Shows

Why Exhibit at IEW 2018? More Technology Decision Makers and Influencers than Any Other Event

India’s Only Test & Measurement Show is Also a Part of IEW

Bag year-end orders: meet prospects in early Feb, get orders before FY ends

It’s a Technology centric Show and Not Just a B2B Show

360 Degree Promotions via Event, Publications and Online!

World’s #1 IOT Show is a Part of IEW and IOT is Driving Growth

3,000+ Visitors are Conference Delegates Alone

Being Held at a Venue (KTPO) That’s Closer to Tech Firms

The only show in Bengaluru in FY 2017-18

Besides Purchase Orders--You Can Bag ‘Design Ins’ and ‘Design-Wins’ too

Your Brand and Solutions Reach Out to 500,000+ Audience, Not Just 15 to 20,000

It’s an Electronics For You Group Property— And We Value Your Trust More Than Money

Co-located Events Offer Cross Pollination of Business & Networking Opportunities

IEW Connects You with Customers Before the Event, At the Event, and even After the Event

Special Packages for ‘Make in India’, ‘Design in India’, ‘Start-up India’ and ‘LED Lighting’ Exhibitors

Why Should You Risk Being an Early Bird? 1. The best locations sell out first 2. The earlier you book—better are the rates, and more are the deliverables 3. We might just run out of space this year!

To get more details on how exhibiting at IEW 2018 can help you achieve your sales & marketing goals,

contact us at +91-9811155335 OR

write to us at [email protected]

EFY Enterprises Pvt Ltd | D-87/1, Okhla Industrial Area, Phase -1, New Delhi– 110020

Insight



 

boxes, labels, menus, buttons and common elements. Physics engine: It supports 2D physics engines like BOX2D and Chipmunk. Audio: It supports sound effects and background music. Network support: HTTP with SSL, WebSocket API, XMLHttpRequest API, etc.

For U & Me

ActionScript 3 library that is very similar to the traditional Flash architecture. It recreates the Flash display list architecture on top of the accelerated graphics hardware. It is a very compact framework but comprises various packages and classes. The following are the sets of tools that are integrated with Starling for application development: Display programming: Every object is a display object. Images and textures Dynamic text Event handling Animation Asset management Special effects Utilities 




Ofcial website: http://www.cocos2d-x.org

     

Features 





Figure 3: Cocos2d-x user interface



Delta Engine



Delta Engine is an open source 2D and 3D app and game development engine maintained by Delta Engine company. The applications and games can be developed in an easy manner through Visual Studio.net or the Delta engine editor. Delta Engine supports various languages and frameworks like C# OpenGL, C# OpenTK, C# GLFW, C# XNA, C# SharpDX, C# SlimDX, C# MonoGame, LinuxOpenGL, MacOpenGL and WebGL. It supports various platforms like Windows OS, OS X, Linux, Android, Android TV and Linux.



Latest version: 2.2

Ofcial website: https://gamua.com/starling

Panda 3D

Features 









It is based on Stage3D and supports multiple platforms like Android, iOS, Web browsers, OS X, etc. It has low conguration requirements in terms of CPU, memory and GPU. It has lower battery consumption. Has effective object organisation via hierarchical trees, i.e., parent-child relationship. Highly powerful and efcient event system using ActionScript. Supports texture atlases, lters, stencil masks, blend modes, Tweens, multi-touch, bitmap fonts and 3D effects.

It supports 3D features like 3D model importing, a particle effect editor, etc. Content like images, sounds, music and 3D models is saved directly using the Delta engine. Supports physical simulation; most code is interchangeable for both 2D and 3D simulation. Supports integration of external libraries and frameworks like the 2D Sprite animation library, Spine. App Builder tool integrated in the editor supports building, deployment and launching of apps on a mobile device.

Panda 3D is an open source framework for rendering and developing 3D games using C++ and Python programs. The entire gaming engine is written in C++ and makes use of automatic wrapper generators to expose the complete functionality of the engine in the Python interface. It supports OpenGL and DirectX. Panda 3D includes various tools like scene graph browsing, performance monitoring, animation optimisers and many more. Features 

 


Ofcial website: https://deltaengine.net



Starling Starling is an open source 2D game development framework that supports both mobile and desktop platforms. It is a pure



Hassle-free installation and supports Windows, OS X and Linux. No need for any sort of compilation. Full Python integration and highly optimised via C++. Comes with various OpenGL and DirectX features like GLSL, a powerful interface between shaders and engine, and supports render-to-texture and multiple render targets. Other features include shader generation, 3D pipeline, support for OpenAL Audio Engine, FMOD Audio Engine and Miles Audio Engine. Has support for the Bullet physics engine, ODE physical engine and PhysX physics engine.


103

For U & Me

Insight

Figure 4: Superpowers’ GUI interface Latest version: version: 1.9.4

Figure 5: MonoGame user interface 

Ofcial website: www.panda3d.org

Superpowers Superpowers is a powerful, open source and free development platform enabling developers to create fully customised 2D and 3D games that are highly exible. It is a cross-platform development tool and supports Windows, Linux and OS X operating systems. It makes use of Typescript Typescript to write gaming logic and to highlight the syntax, which simplifes development. Features 







Easy and well laid GUI interface helps even newbies to quickly learn game development. Has a powerful Typescript editor fully packed with features like syntax highlighting, auto completion of code and live error reporting. Comes with hundreds of inbuilt licence-free sprites, 3D models, sound effects, fonts and music. Built-in library of games and examples acts as a s trong platform for beginners.

Latest version: version: 4.0

Ofcial website: http://superpowers-html5.com

MonoGame MonoGame is powerful free software that Windows based developers and Windows Windows Phone gamers use to run on other systems. It is a cross-platform game development development tool and supports Linux, OS X, Android, PlayStation Mobile, Nintendo Switch, etc. It is basically an open source implementation of the Microsoft XNA 4 framework. The basic objective of MonoGame is to ‘write once, play everywhere’. The following are the technologies that power the MonoGame API’s API’s cross-platform capabilities: OpenTK: A low level C# library that combines OpenGL, OpenCL and OpenAL for 3D graphics. 

104



SharpDX: An open source implementation of DirectX API for .NET, which supports high performance of 2D and 3D games and real-time sound. Lidgren.Network: This is a network library for the .NET framework, which makes use of the UDP socket to provide APIs for connecting to the client and server as well as sending and reading messages.

Features

Via C# and .NET languages, MonoGame enables developers to write reliable and high-performance game code. Open source code enables changes and even porting to new platforms. Bundled with more than 1000 games, MonoGame can be used for high-end games development.







Latest version: version: 3.6

Ofcial website: www.monogame.net

References [1] [2] [3] [4] [5] [6] [7] [8]

http://compilgames.net/ https://godotengine.org http://www.cocos2d-x.org https://deltaengine.net https://gamua.com/starling www.panda3d.org http://superpowers-html5.com www.monogame.net

By: Prof. Anand Nayyar The author is assistant professor in the department of computer applications and IT at KCL Institute of Management and Technology, Jalandhar, Punjab. He loves to work and research on open source technologies, cloud computing, sensor networks, hacking and network security. He can be reached at You can watch his YouTube videos [email protected]. You [email protected]. at youtube.c at youtube.com/anand om/anandnayyar nayyar..

| SEPTEMBER 2017 | OPEN SOURCE SOURCE FOR YOU | www.OpenSourceForU.com

Insight OpenGurus

Communication Protocols for the Internet of Things: A Few Choices With more and more devices being connected to the Internet and millions of devices interacting with each other and the server, the need for communication protocols is critical. MQTT, CoAP, CoAP, and Bluetooth are some of the communication protocols covered in this article.

I

n 2008, the number of connected devices in operation exceeded the number of humans connected to the Internet. It is estimated that by 2025 there will be more than 50 billion connected devices generating a revenue of US$ 11 trillion. Though the term, the Internet of Things or IoT, was rst coined back in 1999, the buzzword has started becoming a feasible reality in recent years. As we can see, the consumer electronics market is already ooded with smart and connected LED bulbs, home automation solutions and intelligent vehicles. Meanwhile, the Do-It-Yourself (DIY) hobbyist sector is seeing ultra-low power and high performance SoCs with built-in Wi-Fi, LoRa or Bluetooth communication features. The prices of radio chips are now as low as US$ 5 and there are tons of new products, unimaginable before but now a reality, as was seen at this year’s Consumer Electronics Show (CES), Las Vegas and Mobile World Congress (MWC), Barcelona. Products like a smart toothbrush that learns your brushing habits, connected drones that can follow and record you while you are in the middle of an adventurous moment like river rafting, or a simple over-the-air (OTA) software update that can turn your car into a smart self-driving vehicle. With IoT and articial intelligence backing it up, the possibilities are endless.

divided into three constituents. 1. The hardware: This makes up the ‘things’ part part of IoT and usually has a small microcontroller with sensors/ actuators and rmware running on it, which is responsible for how it functions. A good example of this would be a smart tness tracker with, say, an ARM Cortex M4 microcontroller and an Inertial Measurement Unit (accelerometers or gyroscopes) sending data to your smartphone via Bluetooth. software: Firmware running on the device, mobile 2. The software: Firmware applications, cloud applications, databases, device management/implementation, the frontend to display data or an algorithm which gives intelligence to your IoT project— all come under the software portion of the IoT stack.

Getting started with IoT IoT provides a great development stack, so everyone can contribute to its development and growth. It can be broadly

Figure 1: IoT - billions of devices connected with each other

www.OpenSourceForU.com | OPEN SOURCE SOURCE FOR YOU | SEPTEMBER 2017 |

105

OpenGurus Insight 3. The cloud: The ability to stream and store data over the Internet, visualise it in a Web browser and control the device remotely from any part of the world is all because of the cloud, which virtually makes the data available anytime, anywhere. There are innumerable ways to get into the IoT space, right away. In this article, I’ll talk about communication protocols for the IoT space, which can be used for communication between machines or between a machine and server. Due to constraints in processing capabilities and the low power requirements of IoT devices (which are generally meant to be deployed in environments with constant battery power) with limited bandwidth capabilities, a need was felt for dedicated standards and protocols especially designed for IoT. Since those who manufacture IoT devices and those who create the IoT platforms are different, this required industry standards and protocols that were not high on power consumption, bandwidth usage, or processing power and could be adopted easily by all IoT players—hardware manufacturers, software developers or cloud solutions/service providers. When developing and deploying an IoT project, it’s important to answer questions like: How do my devices talk to each other or to me? Do I want the stability of a wired network or the freedom of a wireless one? What are my constraints? Data rates, battery power or poor networks? What communication options do I have?  





is overall a lightweight protocol that runs on embedded devices and mobile platforms, while connecting to highly scalable enterprise and Web servers over wired or wireless networks. It is useful for connections with remote embedded systems, where a small code footprint is required and/ or network bandwidth is at a premium or connectivity is unpredictable. It is also ideal for mobile applications that require a small size, low power usage, minimised data packets, and efcient distribution of information to one or many receivers. It is an ISO standard (ISO/IEC PRF 20922) protocol. The good performance and reliability of MQTT is demonstrated by Facebook Messenger, Amazon IoT (AWSIoT), IBM Node-Red, etc—organisations etc—organisations that are using it to serve millions of people daily. An MQTT-SN or MQTT sensor network allows you to use MQTT over a wireless sensor network, which is not generally a TCP/IP based model. The MQTT broker can be run locally or deployed on the cloud. It is further enhanced with features like user name/password authentication, encryption using Transport Layer Security (TLS) and Quality of Service (QoS). MQTT implementation implementation: MQTT can be implemented with a broker and MQTT clients. The good news is that both can be found open sourced in the Mosquitto package, which is an open source MQTT broker available as a package for Linux, OSX or Windows machines. It runs an MQTT broker daemon, which listens for MQTT translations on Port 1883 of TCP (by default). To install it on Debian based machines (like Ubuntu 16.04, Raspbian Jessie, etc), simply run the following command from the terminal:

Enter the world of IoT communications This section covers a list of IoT communication protocols. protocols. 1. MQTT (Message Queue Telemetry Telemetry Transport) MQTT is my preferred IoT protocol, which I use for almost all my IoT automation projects. It was created about 15 years back for monitoring remote sensor nodes, and is designed to conserve both power and memory. memory. It is based on the ‘Publish Subscribe’ communication model, where a broker is responsible for relaying messages to MQTT clients. This allows multiple clients to post messages and receive updates on different topics from a central server known as the MQTT broker. This is similar to subscribing to a Figure 2: The MQTT model YouTube channel, where you get notied whenever a new video is posted. Using MQTT, a connected device can subscribe to any number of topics hosted by an MQTT broker. Whenever a different device publishes data on any of those topics, the server sends out a message to all connected subscribers of those topics, alerting them to the new available data. It

106

# sudo apt-get install mosquito mosquito-clients

This will install and run MQTT broker on your Debian based Linux machine and provide clients the utilities mosquitto_pub and mosquitto_sub, which can be used to test and use it. On the device/client side, Eclipse IoT provides a great open sourced implementation of MQTT and MQTT-SN MQTT-SN version 3.1.1 in the form of a library known as Eclipse PAHO, which is available for almost all modern programming languages like C, C++, Java, Python, Arduino, etc, or can be used over WebSockets. WebSockets. For more details or the API reference, visit http://www.eclipse.org/paho/ . The table in Figure 3 compares HTTP and MQTT, clearly showing why the latter is a winner in the IoT space. 2. CoAP (Constrained Application Protocol) Constrained Application Application Protocol (CoAP) is an Internet application protocol for constrained devices (dened in RFC 7228). It enables constrained devices to communicate with

| SEPTEMBER 2017 | OPEN SOURCE SOURCE FOR YOU | www.OpenSourceForU.com

Insight OpenGurus Characteristics

Receive Message

Send Message

3G

WiFi

HTTPS

MQTT

HTTPS

MQTT

1,708

160,278

3,628

263,314

Percent Battery / Hour

18.43%

16.13%

3.45%

4.23%

Percent Battery / Message

0.01709

0.00010

0.00095

0.00002

Messages Received (Note the losses)

240/1024

1024 / 1024

5 24 / 1 02 4

1 02 4 / 1 02 4

Message / Hour

Messages / Hour

1,926

21,685

5,229

23,184

Percent Battery / Hour

18.79%

17.80%

5.44%

3.66%

Percent Battery / Message

0.00975

0.00082

0.00104

0.00016

Figure 3: MQTT vs HTTP

Figure 4: The CoAP model

Bluetooth

Bluetooth Smart Ready

Bluetooth Smart

Traditional wireless devices, streaming rich content, like video and audio

Devices that connect with both- the center of your wireless world

Sensor devices, sending small bits of data, using very little energy

3. Bluetooth and Bluetooth Low Energy While MQTT and CoAP are infrastructure-independent, which means that it doesn’t matter whether you’re connected to a wired or a wireless network, Bluetooth provides only wireless communication over radio frequency (2.4GHz spectrum in the ISM band) using an industry standard that was initially used to share les between mobile phones and is now powerful enough to play music (Advanced Audio Distribution Prole/A2DP), stream data, or build your next IoT device. Bluetooth, generally, is divided into three categories. Bluetooth Classic: This is meant for high data rate applications like streaming audio wirelessly. Bluetooth Smart or Low Energy/BLE : This is meant for low powered battery-operated devices that stream low packets of data. Bluetooth SmartReady: These are essentially the ‘hub’ devices such as computers, smartphones, etc. They support both the ‘classic’ and ‘smart’ devices. Bluetooth is a sophisticated ad hoc networking protocol, and is now especially designed from the ground up for IoT. It provides a stable connection and communication channel, which is extremely low prole and low powered. An obvious example is tness trackers, which even though powered on throughout the day, can last for months on a single charge or run on a coin cell battery, all thanks to BLE (Bluetooth Low Energy). Bluetooth Classic has xed proles like UART over Bluetooth class and A2DP class for audio streaming. On the other hand, Bluetooth Low Energy provides GATT or Generic Attribute Prole, which allows users to dene their own prole using Bluetooth, like in the case of a heart rate monitor. BLE is extremely exible and useful in the IoT space. Bluetooth 5.0 is already out and is maturing, offering more range, more data rates and double the transmission speeds.

Figure 5: Bluetooth flavours

Which protocol should I use for my next IoT project? the wider Internet using similar protocols. CoAP is designed for use between devices on the same constrained network, between devices and general nodes on the Internet, and between devices on different constrained networks joined by the Internet. It is an application layer protocol designed for network constrained IoT devices like wireless sensor network nodes, and is often termed the lightweight version of HTTP with support for REST APIs. It can run on most devices that support UDP or a UDP analogue. It implements the REST architectural style which can be transparently mapped to HTTP. However, CoAP also provides features that go beyond HTTP such as native push notications and group communication. While a usual HTTP header can be around 100 bytes, a CoAP standard header can be as light as just 4 bytes. Unlike MQTT, CoAP doesn’t require a broker server to function. On the implementation side, the Eclipse Californium project provides a Java implementation of the CoAP protocol, including support for the DTLS security layer. There’s also a MicroCoAP project which provides CoAP implementation for Arduino. Check out https:// github.com/1248/microc.

There are many different protocols and industry standards that are specially designed for IoT or can be used for it, such as the few mentioned above and others like Wi-Fi WebSockets, Zigbee, LoRA, Simple RF, XMPP, RFID, NFC, etc. Yet, one’s choice should be based on the project requirements and the constraints of the application you are thinking of developing. MQTT, for example, is extremely powerful when you have an actuator network which needs to respond to a common sensor. The PUB/SUB model is ideal in that case. In the case of CoAP, you can create your own constrained network environment and relay information to the Internet via proxy. If your project does not require the Internet or long range communication, like a tness tracker, then Bluetooth Low Energy could be the best choice. The possibilities in the IoT space are endless.

By: Ayan Pahwa The author is an IoT, AI and automotive enthusiast working as an embedded software engineer at the Mentor Graphics- A Siemens Business, facility at Noida. He can be reached at Ayan_Pahwa@ mentor.com. Website: http://iayanpahwa.github.io.


107

TIPS

&

TRICKS

Finding out a file’s size before initiating an expensive download With super-fast connectivity, we often downl oad gigabytes before realising that a file is bigger than our budget. To find out the size of the download, install curl and gawk . Then fill a local file sizer.sh with the following commands:

following command: chmod +x ./sizer.sh

Run it with ./siz er.sh and supply your URL to query its size. —A. Datta, [email protected]

#!/usr/bin/env bash echo -n “Enter full download URL (Ctrl-Shift-V to paste): “

A useful tool called Tilda

read FULLURL; RET=0 while [[ RET -eq 0 ]]; do LEN=$(curl -Ls --head $FULLURL | grep -i “ContentLength” | tail -1) TYPE=$(curl -Ls --head $FULLURL | grep -i “Contenttype” | tail -1) grep “text” <(echo $TYPE) >/dev/null 2>&1

RET=$? if [[ RET -ne 0 ]]

We often want to access the terminal to execute some command or to see the progress of some job running inside the terminal; to do this, we need to switch from using Ctrl+tab. For those who dislike switching, there is Tilda. It is a Linux terminal with no borders and is hidden from the desktop till a key is pressed. You can install it using the following command on Debian and similar systems.

then tr -dc ‘[[:print:]]’ <<<$TYPE; echo

sudo apt-get install tilda

echo $(tr -dc ‘[[:print:]]’ <<<$LEN)” bytes”; exit f

DOMAIN=$(echo $FULLURL|cut -d/ -f1-3)

OLDURL=$FULLURL FULLURL=$DOMAIN$(curl -Ls --get -r 0-10240 $FULLURL |

grep -i “http-equiv” | \ grep -i “url=” | awk ‘BEGIN {FS=”url=”} {print $2}’ |

Once installed, you can launch and confgure Tilda by right-clicking on it and then choose Preferences. Assign a less frequently used key combination as a shortcut to launch Tilda so that it doesn’t interfere with your regular shortcut keys. Tilda is very useful when you are following some tutorial and want to code simultaneously while reading it.

cut -d \” -f 1 | cut -d \’ -f 1)

—Amar Shukla, [email protected]

if [ “$OLDURL” == “$FULLURL” ]

then echo “Download size cannot be determined”;

Easily restrict untrusted applications using Linux namespaces

exit f

done

Make it executable once by using the

108

Firejail is a SUID sandbox program that reduces the risk of security breaches by restricting the running environment of untrusted applications. It does this by


using Linux namespaces, seccomp-bpf and Linux capabilities. Once you install Firejail, you will nd it is pretty easy to use. Given below is an example that shows you how to use it. It runs Firefox with the default security proles.

Exec=/usr/bin/idle3

#rejail refox

All the elds of the above conguration are selfexplanatory. Now, you can search the IDLE application in Ubuntu Dash and also lock it to the launcher.

Comment=The Drive to Develop Categories=Development Terminal=false StartupWMClass=IDLE

There are many options available along with Firejail. You can get all the details from the manual page of the command.

—Narendra Kangralkar, [email protected]

—Renjith Thankachan, [email protected]

Adding colours to the BASH prompt To add colours to the shell prompt, use the following export command syntax…

Recording the command session We often type a sequence of commands in Linux and then forget what we have done. Here is a small tip that will help you record all the commands that you type. The recorded commands will be stored in a readable text le named fle1.txt .

\e[x;ym $PS1 \e[m

…where, \e[: Starts the colour scheme x;y: Indicates the colour pair to use (x;y) $PS1: This is your shell prompt variable \e[m: Stops the colour scheme

$ script le1.txt Script started, le is le1.txt

To set a red colour prompt, type the following command:

$ ls $ ps -el $ lsblk $ exit

$ export PS1=”\e[0;31m[\u@\h \W]\$ \e[m “

Script done, le is le1.txt

To set a blue colour prompt, type the following command:

$ cat le1.txt

The output will show all the commands that were executed during the Script session.

$ export PS1=”\e[0;34m[\u@\h \W]\$ \e[m “

— Pritam Nipane, [email protected]

Creating a desktop launcher in Ubuntu Unity Unity launchers are actually files stored in your computer with a ‘ .desktop ’ extension. To create a desktop launcher, create the .desktop file using a text editor and save it under the ~/.local/share/ applications/ directory. For example, given below is the .desktop file for Python IDLE. [bash]$ cat ~/.local/share/applications/idle.desktop [Desktop Entry] Version=1.0 Type=Application Name=IDLE Icon=/media/jarvis/partition/icons/python-icon.png

Here is a list of colour codes: Blue: 0;34 Green: 0;32 Cyan: 0;36 Red: 0;31 Purple: 0;35 Brown: 0;33 —Rajeeb Senapati, [email protected]

Share Your Linux Recipes! The joy of using Linux is in nding ways to get around problems—take them head on, defeat them! We invite you to share your tips and tricks with us for publication in OSFY so that they can reach a wider audience. Your tips could be related to administration, programming, troubleshooting or general tweaking. Submit them at www.opensourceforu.com. The sender of each published tip will get a T-shirt.


109

Open Source for You - September 2017

Recommend Documents

Addition Of Two Numbers

Sorry, there was a problem.