[ Note 0: This became very long.
Note 1: I will try to come back and properly link to the efforts/projects mentioned in this post, but for now, you can "google" most of these and find them successfully.
Note 2: These observations have my usual biases, especially web application testing. Apologies in advance to mobile testing people doing really interesting things. ]
- Google (Ankit keynote) uses "push on amber" rather than "push on green" as a continuous delivery philosophy (if I understood correctly, the primary motivators are that this allows for a "push decision", exploratory testing, and other manual stuff to be in the loop, but to be informed by test results).
- A "hermetic test" is one that could run on a (Wifi-disabled) airplane, with no network. Such tests should be the goal of teams moving fast.
- We're going to present MSL tomorrow ... we use it to achieve this on UI tests at FINRA. When this was mentioned, there was much pumping of fist on our row.
- Google (Ankit keynote) advocates a testing philosophy antithetical to the "ice cream cone" anti-pattern (see here). You can see the same idea in our MSL preso here. Again, much pumping of fist when this was mentioned. May have even been an "Amen".
- Google has a tool called Sheriff for identifying and AUTOMATICALLY REMOVING flaky tests.
- James Graham and others at Mozilla are working on tests for browser interoperability (via the W3C Open Web Platforms standard, a work in progress). To be somewhat neutral (I guess), they've developed their own client-side test framework (looks like *Unit, in JS). They also developed a Runner, and an "expectation file format" which tells a web server what to serve up for a particular test.
- Karin Lundberg's talk (although focused on testing Mobile Chrome) brought up some interesting general testing advice:
- Developers tend to prefer using the same language for test dev and feature dev.
- When introducing new test frameworks, interesting examples and plans for utilities are important.
- Chrome has a performance testing tool called Telemetry.
- Tests (always) need to be able to run in tool-agnostic but tool-compatible way.
- Google did a broad analysis of the code coverage metrics on its projects and came up with a standard of 85%. They also have enhanced their code review tool with unit test coverage numbers.
- I did not handle this talk well as I do not care for isolated discussions of code coverage. It is necessary (obviously you can't find a bug that you don't cover); but it is not sufficient. Yet somehow, it was the only test quality metric that got much attention today. How can we possibly have a uniform standard for code coverage without critical standards for other metrics right beside it?
- [ To be clear, the presenter today wasn't suggesting anything about the "sufficiency" of coverage. The extension of the code review tool is interesting, and code coverage is important. ]
- Cat.js is a tool for converting annotations in mobile application code into test cases that can run on real devices.
- Vishal from Dropbox explained how they are abandoning Jenkins CI in favor of a largely homegrown CI solution called Changes. [I see no changes ... wake up in the mornin' and I ask myself ... ] The motive is better support for sharding of tests and running things in parallel.
- For resource management and allocation, they use Mesos on top of LXC
- [ Speaking of LXC, no one mentioned Docker today ... ]
- One thing that wasn't mentioned was whether app instances were being duplicated or just test driver machines (e.g. Jenkins' "slave nodes"). If each test does its own setup of data, does it not need its own app container, 1:1?
- Gareth Bowles from Netflix talked about all of the testing that they do IN PRODUCTION.
- The coolest thing to me was the fact that they have coverage enabled in production. They use Cobertura, which does add an overhead of around 5% on their production systems. This is probably because Cobertura doesn't work via dynamic instrumentation but static (you have to generate an instrumented version of the source code and add this to the classpath for data to be available). He did mention that they have extensions to their base AMIs, Tomcat installs, etc. to make this relatively easy to enable.
- One interesting problem at Netflix's "Internet scale": There is no possibility for a "QC environment with prod-like volume", because prod-like volume is HUGE (a significant chunk of all US Internet traffic, for example)
- They have various "monkeys" which can bring down instances (Chaos Monkey), and simulate the failure of availability zones and entire regions of service in AWS. And they run these tools in production (giving warning for the latter two, though the instance termination is on by default in production).
- Jay Srinivasan's talk on an infrastructure for real device testing described a real device infrastructure and accompanying test runner:
- It can (obviously) target tests to a given set of devices
- It can simulate network latency and locality issues
- It also comes with a "robot" for systematically exploring the application space through UI events [ this reminded of the Android GUI Ripper work Atif did with the group from Naples ].
- Also from Jay's talk, a study of 1-star mobile app reviews found that around 40% of underlying issues related to crashes and functional bugs (not the harder-to-test performance issues, or even "usability" or something else fancy).
- Celal Ziftci presented a machine learning tool which can infer invariants from analysis of production logs and enforce these in real time (in production, after some optional filtering).
- From what I understood, at Google, they use protocol buffers to enable somewhat standardized logging of serialized objects at critical points in the system (and especially between systems).
- They can also compare Prod invariants to invariants picked up during test to gain some confidence (or not) about the usage coverage of testing.
- Nan Li is one of Jeff Offut's former students at GMU who created a language for providing mappings from state machines of applications to "concrete test cases" (which can be executed).
- Of course I love model-based testing, but one of the main reasons my academic advisor invented GUI Ripping 10 years ago was because constructing state machines is very complex for any non-trivial application. Perhaps on constrained domains (or ridiculously important ones like space ships) this is worth trying.
- Nan will be applying these techniques at Medidata Solutions, doing something with genomics. It may be a great fit, but I don't see it scaling to web apps.
- Orange is using Raspberry Pis to help automate testing of real set-top box devices.
These are my notes. What are yours? What did I get wrong?
The information you have deliver here is really useful to make my knowledge good. Thanks for your heavenly post. It is truly supportive for us and I have accumulated some essential data from this blog.ReplyDelete
Software training|Software testing training in chennai
Great and useful post.Keep on sharing.ReplyDelete
Manual Testing Training institute in Chennai | Manual Testing Training | Manual Testing Training in Chennai
Awesome post, you have explained it very well.ReplyDelete
Thanks a lot for a great article.
Selenium training in Chennai | Best Selenium training institute in Chennai
This is an extraordinary article. It will help to improve my knowledge. Thanks for sharing this useful blog.ReplyDelete
Software Testing Training Institutes in Chennai | Best Software Testing Training Institute in Chennai
whatsapp unblock myself software
lady to obtain 10kgs more for rs.100, find the original price per kg?
about bangalore traffic
how to hack whatsapp ethical hacking
the lcm of three different numbers is 1024. which one of the following can never be there hcf?
how to hack tp link wifi
whatsapp unblock hack
sample resume for call center agent for first timers
a merchant sold an article at 10 loss
very nice post...ReplyDelete
internship report on python
free internship in chennai for ece students
free internship for bca
internship for computer science engineering students in india
internships in hyderabad for cse students 2018
electrical companies in hyderabad for internship
internships in chennai for cse students 2019
internships for ece students
inplant training in tcs chennai
internship at chennai
very nice post.........ReplyDelete
r programming training in chennai
internship in bangalore for ece students
inplant training for mechanical engineering students
summer internships in hyderabad for cse students 2019
final year project ideas for information technology
bba internship certificate
internship in bangalore for ece
internship for cse students in hyderabad
summer training for ece students after second year
robotics courses in chennai
3d modeler resume samples
good... nice... very useful..ReplyDelete
it is best blogs ....ReplyDelete
good .........very usefulReplyDelete
Your article is very informative. Thanks for sharing the valuable information.ReplyDelete
DevOps Training in Chennai
DevOps Training in Bangalore
DevOps Training in Coimbatore
Best DevOps Training in Marathahalli
DevOps Training Institutes in Marathahalli
DevOps Institute in Marathahalli
DevOps Course in Marathahalli
DevOps Training in btm
DOT NET Training in Bangalore
PHP Training in Bangalore
Thank you for sharing such great information with us. I really appreciate everything that you’ve done here and am glad to know that you really care about the world that we live in.ReplyDelete
Microsoft Windows Azure Training | Online Course | Certification in chennai | Microsoft Windows Azure Training | Online Course | Certification in bangalore | Microsoft Windows Azure Training | Online Course | Certification in hyderabad | Microsoft Windows Azure Training | Online Course | Certification in pune
Movie-watching websites that are more than movie-watching websites Because we are the number 1 free movie site in Thailand for a long time, including new movies, Thai movies, Western movies, Asian movies, we have all kinds of ways for you Including new series Full of all stories without interstitial ads to keep annoying anymore. One place sa-movie.com.ReplyDelete
Android and IOS operating systems. Watch online movies, Thai movies, Western movies, Asian movies, Cartoon movies, Netflix Movie, Action Movies, Comedy Movies, Crime Movies, Drama Movies, Horror Movies, Adventure Movies, Crash Movies and still have many new movies to watch. You can watch for free anytime, anywhere 24 hours a day at see4k.com.
GangManga read manga, read manga, read manga online for free, fast loading, clear images in HD quality, all titles, anywhere, anytime, on mobile, tablet, computer. Android and IOS operating systems. Read top comics, action dramas, comedy, adventure, horror and manga. New coming every day to watch many more. Can be read for free anytime anywhere 24 hours a day at gangmanga.com..
It is no secret that football is among the most popular and widely watched sports. Everybody who likes football tries to find the best platform for free soccer streaming. So, what are the best free sports streaming sites? We are going to answer this question. On this page, you can find a detailed overview of the most widespread soccer streaming websites. Keep on reading and make the best choice for you live24th.me.
Nice post. I learn something totally new and challenging on websites. I stumbleupon on a daily basis. It will always be exciting to read articles from other writers and use something from their Websites. ยูฟ่าสล็อตReplyDelete