In late 2010, we had a mixed environment: Ruby for our web app and C++ for our firmware and performance-sensitive backend daemons. So everyone was familiar with how easy and enjoyable it was to write code in Ruby, but also keenly aware of the performance penalty for doing so. When we heard that Twitter was using Scala, we took a second look, and it seemed like a nice combination of the expressiveness of Ruby while retaining performance much closer to C++. After a few small experiments to play with the language, we used Scala for the next big backend daemon we wrote, and we were hooked. We've been using Scala in production since early 2011.

We use sbt, of course. We use Scalatest for tests, and Jacoco to check test coverage. Some of us use Eclipse, but others still prefer vim. And we use all the usual tools built into the jdk for heap dumps, tracking garbage collection frequency, debugging, etc. They aren't exactly "tools", but having the wide variety of java libraries available has also really helped with our Scala adoption.

We use TeamCity for continuous integration. For deployment, we build a .war file and hot reload it into jetty.

We're explicitly against having any one main programming style. For some programming problems, functional programming describes a solution much more cleanly than any other methodology, but for others, an object oriented approach is more clear. The same can be said of threads versus events versus message passing. We try to pick the best approach for each individual problem that we solve.

The keynote sounds great. The type theory nerds among us are looking forward to Jared Roesch's Shapeless talk. Most of us are excited to hear about how other companies are using Scala. It's wonderful that the ecosystem is growing so quickly!

Honestly, the content is already quite good.

We use it more and more every day. We have a large Rails app to maintain, so still it makes sense for us to write some new code in Ruby, but any time we know something is going to be performance critical, we write it in Scala.

Our core team joined PayPal in December as a part of the acquisition of StackMob, a backend-as-a-service provider, which took place this past December. We brought our experience as Scala developers to the company, and have been developing new services in Scala since December. We learned that there were several pockets of interest in Scala at PayPal, including a still-developing Scala internal framework meant to help migrate existing Java developers and projects, but to my knowledge we are the first full-fledged Scala team at the company.

We build services primarily using Spray and Akka. We had prior success with Spray, and we find ourselves quite comfortable working with its patterns. One notable difficulty we have encountered with Spray/Akka is correctly tuning our stack for a heterogeneous platform; after experimenting with several profiles, we saw drastically improved performance! We also pursued a Typesafe subscription, which we hope will be a continued resource for us going forward. In addition to Spray and Akka, we make extensive use of sbt for project definitions, Scalastyle for code style enforcement, the Scala-friendly code generators in Swagger for API definitions and consistent clients, and Specs2 and ScalaCheck for unit and integration testing. We are also investigating Scoverage as a potential augmentation to our existing JaCoCo coverage suite.

Our team runs our own CI stack - notably Jenkins and Sonatype Nexus - which integrates with our internal GitHub. Our Jenkins also facilitates a Gatling load-test suite, that tracks performance throughout the day in our main development line. This is all separate from the substantial layer of company-wide integration and regression testing that takes place regularly. Thanks also to Scala-Java interop, we are also tied into several company-level monitoring systems, to which we as developers have limited access. We are pushing for more fine-grained alerting, monitoring, and reporting, as we push the limitations of existing internal technology.

Our team takes a hybrid approach to Scala, using FP patterns where appropriate and Java interop where necessary. As we are a singular Scala team, we have no outside Scala patterns to reconcile. We have a team-maintained code style document that helps enforce internal consistency, and we defer to the Scala-lang style guide frequently. We also prefer to avoid frameworks where possible, and as such we have made the active choice to not use the aforementioned internal Scala framework, instead choosing libraries as we need to help keep our packaged code small.

Scala at PayPal is still decidedly a minority language, as compared to Java and C++. We have the benefit of Scala's interoperation with Java to drive traction, however, which has greatly eased our integration into the PayPal ecosystem. Recently, our team helped to pilot an internal Scala training course, which was a success! We are now helping to pilot an advanced Spray/Akka course in similar fashion, and we hope that these will help individual contributors and team members to drive adoption on their respective projects. Our team also acts as a point of contact for others inside the company looking to explore Scala.

How do I choose, when they all look so great?? Greg Soltis' talk on highly concurrent heterogeneous systems and James Earl Douglas' talk on Scala devops are both directly applicable to our daily life at PayPal. Kelsey Gilmore-Innis and Stew O'Connor's talk on ScalaCheck looks as though it will be at least as mind-blowing as Ricky Nilsson's talk at Scala Days (and that was a seriously mind-blowing talk - definitely go see it, if you haven't yet). Li Haoyi's talk on cross-JVM/JS development echoes much of the current PayPal development landscape, which should prove interesting. Ivan Topolnjak's Kamon talk is right in line with what our team is investigating for metrics. And who knows what will come up during the Unconference session?

Hard to say. Alexy and the rest of the SF Scala group are pretty great about including a wide range of topics, geared toward a wide audience of skill levels, in the SF Scala meetup track. It comes as no surprise to me that so many of their talks fill to capacity! I'm also quite satisfied with the conference lineup as-is. Keep up the great work! I suppose I personally would like to see a talk on Scala development for Android, or Scala in graphics or game development. I know of Nathan Hamblen's Spde, a Processing-like tool for teaching graphics, but its latest GitHub activity was two years ago, so I don't suppose it's seen much love lately.

Back in 2012, a team at Intel started working on a green-field project to build a next-generation TV service from the ground up - everything from the silicon to the service platform. From the very beginning the service platform was built using Scala and functional programming practices; its been Scala since day one. The entire team was later acquired by Verizon in early 2014 and we continue our work on that very same service platform today.

We use a lot of Scalaz, Scalaz Stream, Shapeless and Akka (along with their HTTP counterparts Spray and Play). Having a generic programming toolkit like Scalaz (along with Shapeless HList etc) has allowed us to build software that is - for the most part - correct by construction: if it compiles, it works correctly at runtime. We have had difficulties with certain libraries over time, particularly where binary compatibility is concerned, but the overall positive impact of using those libraries has eclipsed any negative aspects.

Whilst Jenkins is the centre of our automated build and deploy pipeline, we have extended SBT fairly significantly to provide a unified mechanism for building and releasing services as immutable servers. "Baked" machines then automatically get deployed to the development environment, and automatically promoted to QA if the test suite passes. All deployed services are automatically monitored with our in-house solution based entirely on Scalaz Stream, which automatically re-partitions monitoring load inside its own cluster, depending on the amount of systems currently being observed.

There are of course some expert functional programmers, along with folks who only started F.P. a few months ago, which is clearly quite a spectrum - in that frame we tend to focus on broad productivity and keeping a maintainable set of functional core libraries, which all team members consume. By doing this it minimises the cross-cutting technical debt and isolates our "learning FP debt" in a specific service (which is far easier to refactor over time than a library that is scattered through lots of subsystems). In addition, all code receives a review before hitting the mainline release branches and we encourage pair programming where possible.

ICFP/CUFP 2014, NEScala 2015, Scala Days 2015

Large scale production deployments with information about scaling FP education in teams; pain points observed by real companies during real business.

Scala has a bright future within Verizon OnCue: we have built a lot of awesome technology using Scala and we are immensely proud of what we have done to date - long may it continue!

VigLink has been a Java and Ruby shop since its inception in 2009. Over the years there were attempts to introduce Scala that didn't stick. Our monolithic Java codebase didn't lend itself to experimenting with new technologies. In 2013 we made significant progress towards breaking that monolith into smaller components, which opened the door for Scala.

Our revenue feeds processing system was the first component chosen to demonstrate some of Scala and FP's strengths. Once that first system was baked, we felt confident using Scala more broadly.

Projects using Play, Spark, and Epic are currently in development. The data science team recently looked at both iscala-notebook and scala-notebook. As for IDE's, we use both Eclipse and IntelliJ.

The most difficult choice was the http framework. Last year we built an administrative web-service in Spray. After it was announced that Spray would be incorporated into Play, Play became the most attractive option due to the larger set of documentation and available support from Typesafe.

Referential transparency works as advertised. The benefit is most noticeable during design and debugging discussions that happen over HipChat with our remote team in Lima, Peru. It's a very effective way to build software.

Jenkins builds jars and wars from our source code in Github, and publishes them to our Nexus artifact repository. Our production system is all AWS. We use Puppet for configuration management, and Rundeck to manage deployments. We use several monitoring systems including New Relic, Pager Duty, Nagios, Cacti, and Graphite. For testing, we use Specs2 and Mockito. We have just started to use ScalaCheck for some property-based testing.

The portion of our 17 (and growing fast!) engineers that work with Scala isn't quite big enough to support many Scala "tribes". As the team gains experience with the language and ecosystem, we will likely converge on small number of VigLink-preferred styles. Code that interacts with our existing Java will retain the OO style, whereas newer code will be more functional. Creating a standard Scalastyle file is on our "to do" list. Test coverage is our most important source code metric.

Irrespective of language, we try to set aside an hour each week for code walk-throughs and tech talks. This is especially useful when using any new library or tool, or when we're refactoring complex old code that hasn't been touched in a while.

We recently prototyped real-time log analysis using Spark, so Matei's keynote on Saturday ranks high among the talks we're anticipating. Between the seven VigLink engineers attending Scala by the Bay, we're interested in all the talks.

I'm personally interested in how Abstract Algebra has been popping up in so many Scala projects, including Algebird and Spire. I'd also like to have a better handle on how far Scala's dependent types can go and have more hands-on experience with Shapeless.

VigLink and the SF Scala community could benefit from more discussion about higher-level design principles. For instance: beyond the mechanics of ad-hoc polymorphism and Scala's context bounds, how are software systems that employ this machanism organized?

A talk on performance tuning including tool demonstrations would be very useful as well.

We hire engineers that have experience or interest in a range of languages and technologies. That creates a receptive engineering culture. Architecturally, the key for us has been factoring our monolith into smaller components that can be modified independently. Of course there are many other good reasons to do this that have nothing to do with Scala, too.

Scala and Java will continue to live together in harmony at VigLink for many years to come. There so much innovation coming out of the Scala community that we can't ignore it. For VigLink that means asynchronous http frameworks, natural language processing libraries, and "big data" analytics libraries and tools. The tactical advantage of staying on the JVM makes Scala significantly less costly and risky than alternatives.

Nitro Cloud was already running on the Play 1.2 Java stack. We got hooked on the reactive principles and adopted Scala and Play 2.2 to build a multi-client, scalable document processing and collaboration platform. This was an internal grounds-up initiative based on lessons learned with Play 1.2 Java stack.

We use SBT, Play, Slick, Akka, Scalaz, and have built our own Scala Async S3 client. Some tools are particularly powerful yet difficult to onboard because of deep domain specific intricacies - SBT especially. Slick was an interesting transition from Java+Hibernate world as certain concepts don’t map over at all.

Our Platform tests are fully automated with the Play built in testing support and specs2 framework. We do Continuous Integration and every commit gets built by our CI environment which runs the tests, packages a release and deploys to the Developer Integration Environment. We also do automated on-demand stress tests leveraging this stack.

We focus on being pragmatic. We run self-organized teams across different projects - and we spread learnings from project retrospectives through internal tech talks where engineers potentially pitch for a change in process to the team. This lets us practice our presentation and pitching skills and keeps us honest and disciplined in letting the business needs drive the innovation. We have open-invitation team-wide design and architecture reviews. We learn from each other on best practices and designs in building a robust and scalable Scala architecture. In general, based on core Scala levels from http://www.scala-lang.org/old/node/8610, we have a few experts with previous large-scale Scala experience who are A3/L2 and the rest are Java experts at the A2/L1 level - we’re internally great at recognizing this and pairing up organically.

Scala by the Bay of course! We also regularly attend SF Scala, Play and other related meetups. We’ve hosted a Play Meetup ourselves, sponsored public Play hackathons in past. We aim to stay involved in the community!

We are primarily looking to meet other Scala enthusiasts and learn from them. Whether it’s best practices, frameworks, or new Scala features we want to evaluate them and learn from others. We believe in continuous learning and incorporating proven coding or process practices that help improve ourselves and our product.

We started with a small team of senior Java engineers and paired them with a few Scala experts new to the company. During this fused onboarding to Scala we recorded lessons learned and then spread the learning and the best practices to the rest of the team. Now most of our engineers can comfortably contribute to the Scala codebase and the central part of our cloud architecture runs on Play 2.2 Scala. We are bullish in spreading the reactive stack to the rest of our products and services. We expect to adopt Play 2.3 sbt-web for all our sites and move our entire back-end codebase to Scala in 2014. At the core of Nitro’s secret sauce is more than 500 KLOC of good old faithful C++responsible for document rendering, format conversion, stamping, and more. We’re next leveraging state of the art reactive Typesafe stack comprised of Akka actors to scale to millions of documents per day with fault tolerance, resilience & high performance.

Scala is central to our story and permeates our recommendation, personalization, ads, and core services. Two software engineers at StumbleUpon were fond of functional programming and not so fond of Java. They used Scala to build two very successful products (content addressable store on top of HBase and a badge server) that are still actively used today. These products were built in short time and with very few engineering resources. These projects served as a great testament to Scala’s scalability both in terms of performance and developer productivity. After this success Scala was adopted by the advertising team to rewrite the ad server from scratch and then more recently we have pivoted toward a microservices approach with Akka.

We use Akka, Spray, Slick, Spire, Shapeless, ScalaTest, ScalaCheck, SBT, Finagle, and Scalding to operate at our scale. Instead of coding concurrency, we enjoy being able to let our code run concurrently as long as we adhere to correct functional programming principles.

Benefits: We can be more productive by writing succinct and expressive code in Scala. Concurrency is much easier to obtain. FP makes code more robust and reusable.

Difficulties: Onboarding new developers not familiar with Scala. Lack of tooling for Scala.

This is an area under active work as we transition to a continuous delivery approach using the Atlassian stack, cloud-based deployment stacks, Docker, and begin disciplined adherence to TDD principles.

We prefer a pragmatic approach where the team will choose their modus operandi as long as the team is able to accept the risk, correctly balance between performance requirements and code complexity, and assume the task of educating new team members. This results in both the functional-centric and Java-esque camps.

Designing Highly Concurrent, Multi-Protocol, Multi-tenant Services in Scala

Scala devops

Building Microservices with Scala

ScalaCheck

Reasoning with Types

Demystifying Shapeless: An Exploration of Dependent Types in Scala

Data as the Killer App for Functional Programming

Longer deep dives into complex topics. These might include advanced Akka patterns such as event sourcing and using circuit breakers, macros, building recommender systems, NLP, etc.

We have had strong Scala advocates and evangelists that have advanced its use and proved its utility and strength for production-quality systems. This has paved the way for other teams to begin their own discovery of what Scala and its ecosystem can bring to them. With the support of management, we are witnessing increased adoption of Scala for new work and working to forge relationships with key players in the Scala community such as Typesafe. We both hire seasoned Scala professionals and are also willing to provide key training for all interested employees. We expect the adoption trend to continue as our pool of Scala talent grows

Twitter has been using Scala since at least 2009. While it was still comparatively early, it was seen as a viable alternative to Ruby, the chief programming language at the time. Twitter’s Scala usage came into full bloom during 2010, as we began to break up our Ruby monolith into a larger set of services. The origin story of Scala at Twitter has been recounted previously [1].

Most of our software infrastructure is homegrown. We use our build system, Pants [2], our own systems software stack [3, 4, 5], our own Thrift compiler [6], large-scale data analytics tools [7, 8], and .. well you get the point. This isn’t to say that we don’t make significant use of all the excellent work happening in the Scala community (to which we hope we’ve contributed by open sourcing large swaths of our own infrastructure). Most significantly, we use ScalaTest and ScalaCheck for most of our testing. Many engineers at Twitter also use the various IDEs available for Scala.

By this, I assume you mean: how do we mix operations and development? I’d say that most engineers are involved in production. There’s no wall over which we throw code. While it varies a bit by team, we have engineers who are dedicated to focus more on operational aspects of the software we run. We call them Software Reliability Engineers, or SREs. The SRE role is a mixture of a traditional developer role and operations role. Their job is primarily to make our software more operable. That said, operations is everyone’s responsibility, and ‘traditional’ engineers carry pagers (or at least iPhones) just like our SREs do.

Certainly we try not to. In a large organization with a lot of code, it’s important to retain a certain degree of uniformity. Code should feel familiar, use the same idioms. We put a lot of focus and effort into enhancing code readability. Not just readability in general, but readability for the average Twitter engineer. This often precludes using more cutting-edge and esoteric techniques, but we believe we’re better off eschewing possible benefits for greater understandability from a larger group of engineers.

Also, it’s important to not forget that we’re building systems. If our use of a language gets in the way of that, then there’s something wrong.

SOSP. I love the vibe and energy from that crowd.

I would like to see more discussion about how to introduce Scala, and FP more generally, into large organizations. What works, what doesn’t? I would also love to see more quantifiable experience reports. How do we know what we’re advocating is actually working? How do we know it’s good?

Our Scala adoption is already very large, and only growing. Scala is a very valuable tool in our relentless pursuit of systems building. Not only do we use it widely, but I feel like we do a good job of using its various strengths. Projects like Finagle and Scaling are both widely useful, and they each exploit different strengths of Scala as a language. If anything, our experience has been that Scala has been a very versatile tool, and I’m sure we’ll continue to exploit that.

We use Scala in anger, as they say.

We had a Java application that was becoming a burden to maintain, so we switched to Scala and never looked back.

Mostly REST APIs using Dropwizard (not technically scala, but works nicely). We are moving from Maven to SBT. We used Scalding for our batch processing at Identified and it was great, though we did hit a few serialization issues using Scalding with Avro. We are now moving towards Spark for batch processing at Workday for performance and security reasons.

We have a dedicated devops team at Workday which rotate our developers in and out of. They manage a VMWare cluster using Capistrano + Puppet. Testing with scalatest on Jenkins. We force ourselves to have 100% code coverage with combined integration and unit tests. To help work with our SOA for development and testing, we have a small project that knows how to start/stop any of our services.

Yes. At Workday integrations team seems to be more FP-extreme while our team is more Java-like. Overall, we try and find what fits the problem at hand best while still leaving the code understandable for new people to pick up.

Pretty much anything that deals with data engineering.

I feel like I have a good grasp on the basics of Spark/Scalding, but I would love to see best practices and real world examples of data pipelines built with Spark or Scalding. Not just word counts ;)

We are slowly trying to convert more and more of the Workday stack over to Scala. Nobody seems to mind :)