Wyniki ankiety: TestNG Survey 2012

  • 1 strona

  • I'm about to take 3 minutes of your life. Are you willing to sacrifice them? :)
  • 1. Why do you use TestNG

    Pokaż odpowiedzi Ukryj odpowiedzi
    Odpowiedź   % Liczba odp.
    It offers features I need
     
    67,74%
    42
    It supports more than unit tests
     
    62,90%
    39
    It is actively developed
     
    35,48%
    22
    It has great community
     
    20,97%
    13
    My team uses it
     
    22,58%
    14
    Other
     
    6,45%
    Wypełnienia: 62
  • 2. Which version of TestNG do you use?

    Pokaż odpowiedzi Ukryj odpowiedzi
    Odpowiedź   % Liczba odp.
    Always the latest one
     
    50,00%
    31
    Always the latest beta version   0,00%
    0
    6.x
     
    53,23%
    33
    5.x
     
    8,06%
    5
    4.x   0,00%
    0
    I don't know
     
    3,23%
    2
    Wypełnienia: 62
  • 3. Which language you use TestNG with?

    Pokaż odpowiedzi Ukryj odpowiedzi
    Odpowiedź   % Liczba odp.
    Java 7
     
    51,61%
    32
    Java 6
     
    79,03%
    49
    Java 5
     
    16,13%
    10
    Java 1.4
     
    1,61%
    1
    Scala
     
    4,84%
    3
    Groovy
     
    9,68%
    6
    Other
     
    1,61%
    Wypełnienia: 62
  • 4. How do you run TestNG tests?

    Pokaż odpowiedzi Ukryj odpowiedzi
    Odpowiedź   % Liczba odp.
    From my IDE (Eclipse, Intellij IDEA)
     
    61,29%
    38
    Using Eclipse TestNG plugin
     
    40,32%
    25
    With Ant
     
    14,52%
    9
    With Maven
     
    80,65%
    50
    With Gradle
     
    9,68%
    6
    From command line (testng.jar + testng.xml)
     
    6,45%
    4
    Other
     
    6,45%
    Wypełnienia: 62
  • 5. Which features of TestNG do you use?

    Pokaż odpowiedzi Ukryj odpowiedzi
    Odpowiedź   % Liczba odp.
    Factories
     
    21,31%
    13
    Groups
     
    67,21%
    41
    Dependent methods / groups
     
    55,74%
    34
    Annotations transformers
     
    16,39%
    10
    Custom listeners and reporters
     
    32,79%
    20
    Dependency Injection
     
    29,51%
    18
    Emailable report
     
    8,20%
    5
    Rerunning of failed tests only
     
    31,15%
    19
    Parameters (from testng.xml)
     
    31,15%
    19
    Data providers
     
    77,05%
    47
    Parallel tests execution
     
    44,26%
    27
    New Assertion
     
    14,75%
    9
    Other
     
    1,64%
    Wypełnienia: 61
  • 6. Which types of tests do you run with TestNG?

    Pokaż odpowiedzi Ukryj odpowiedzi
    Odpowiedź   % Liczba odp.
    Unit tests (testing classes in isolation)
     
    85,48%
    53
    Integration tests (e.g. with Spring / Hibernate)
     
    66,13%
    41
    End-to-end tests (e.g. with Selenium)
     
    45,16%
    28
    Other
     
    6,45%
    Wypełnienia: 62
  • 7. Features you wish TestNG had...

    Pokaż odpowiedzi Ukryj odpowiedzi
    Odpowiedź   % Liczba odp.
    - Reliable execution order (to avoid overlapping of conflicting setups/teardowns) - Better diagnostics (i.e. ant timout silently kills test. There are many cases when TestNG simply gives no hint about the issue on the CLI) - Nice, better default reports - Better documentation, nicer website
     
    8,33%
    1
    I wish I could run my Cucumber-jvm tests with it instead of having to use JUnit.
     
    8,33%
    1
    Better support for validation aspects that cross-cut test cases. This can be mimics using @AfterMethod and ITestResult to set the test to a failure if an additional assertion occurs. However that has races with the reporters due to bugs. This is useful for validating that nothing ended up in a bad state for complex cases (e.g. transactions in a graph database). @DataProvider being run prior to @BeforeMethod and using mocks surprises new engineers a lot. This is a bit painful for code with mocks. Better ability to setup test fixtures. I really love working in Scala specs2 due to how easy it is to create different fixtures for tests within a single class. JavaDoc! Its really poor, but the user guide is pretty decent. More internal tests. Bugs appear in releases, are reported, and often ignored. Its surprising how little TestNG has unit tests for its own behavior.
     
    8,33%
    1
    I wish it had less features, as several of the existing ones are rarely used and of dubious value. Also, a rewrite of its implementation code would be useful to me, since it's very hard to understand its internal workings (when compared to JUnit); would make it easier to integrate with other testing tools (mocking APIs, for example) and probably with IDEs as well.
     
    8,33%
    1
    A better way to order tests. Some modules run in the declared order and others appear to be random. I had one group within a class that needed extra data to run, so I used Group to set that up, but some tests outside the group ran interspersed with test within the group. Luckily for me those tests were not affected by the extra data, but they could have been.
     
    8,33%
    1
    Combined data providers. E.g. somehow easy defined data provider which is in fact cartesian product of other, existing data providers. Or use two data providers in test, like for example first parameter is from first DP, second from second DP, etc. Something like that.
     
    8,33%
    1
    More 'mature' Eclipse plug in (currently have no function to load all 'unit tests' or 'all integration tests' based on a Maven style setup (file patterns and/or annotations)
     
    8,33%
    1
    Possiblity to turn on a feature to have one test class instance per test method (like junit)
     
    8,33%
    1
    Faster and lower footprint reporters.
     
    8,33%
    1
    - a default group for tests with just @Test annotation (to not miss some of them when include/exclude specific tests when running just fast unit tests and/or slower integration tests having different groups)
     
    8,33%
    1
    1) better documentation: e.x. ready-to-use xml snippet for surefire plugin integration to execute multiple groups on different executions or how to run both testng and junit tests in one maven build. 2) something like runners in JUnit. I'd love to get rid of following code snippet in every test @BeforeMethod public void before() { MockitoAnnotations.initMocks(this); } to be able to use @Mock from Mockito 3) Be able to use many data providers in one test method. It's useful for large collections of data sets which can be logically separated but for which code behavior is the same and can be tested by the same test method.
     
    8,33%
    1
    Implicit method dependencies/ordering of test methods using simple annotation on a class, possibly with a customizable regex to allow custom ordering without needing complicated listeners/transformers. Behavior would be exactly like dependsOnMethods but not require every test method be annotated bit could be triggered at the class level.
     
    8,33%
    1
    Wypełnienia: 62
  • 2 strona

  • Now I would like to ask few questions about other tools that you use to create and maintain your tests.
  • 8. Is TestNG your primary testing framework?

    Pokaż odpowiedzi Ukryj odpowiedzi
    Odpowiedź   % Liczba odp.
    Yes
     
    82,26%
    51
    No
     
    17,74%
    11
    Wypełnienia: 62
  • 9. What other testing frameworks do you use?

    Pokaż odpowiedzi Ukryj odpowiedzi
    Odpowiedź   % Liczba odp.
    JUnit
     
    85,37%
    35
    Spock
     
    12,20%
    5
    Scala Test
     
    19,51%
    8
    Groovy Test (language testing features)
     
    12,20%
    5
    Unitils
     
    4,88%
    2
    JBehave
     
    17,07%
    7
    EasyB
     
    2,44%
    1
    Cucumber
     
    14,63%
    6
    Other
     
    14,63%
    Wypełnienia: 41
  • 10. Why do you use other testing frameworks?

    Pokaż odpowiedzi Ukryj odpowiedzi
    Odpowiedź   % Liczba odp.
    For maintenance of legacy code which uses this frameworks
     
    4,55%
    1
    I never needed the extra features TestNG has over JUnit. Also, IDE integration for JUnit works much better, at least in IDEA.
     
    4,55%
    1
    JUnit prior to my arrival and used for integration tests. I switched the team to TestNG and introduced modern practices. They love it! Introduced Scala, where we use specs2. Its far more natural, a pleasure to work in, and can better handle complex test cases. Our front-end uses node.js where we use mocha. Just different toolchain.
     
    4,55%
    1
    Easier, more reliable, more standard.
     
    4,55%
    1
    Lot of Features are than when compare to other tools.
     
    4,55%
    1
    Because they have better support for frameworks in which my applications are written in (Android, RoboGuice)
     
    4,55%
    1
    To support other teams that don't use TestNG. For BDD design.
     
    4,55%
    1
    Because the DSLs are less verbose.
     
    4,55%
    1
    JUnit is simpler
     
    4,55%
    1
    Right tool for the job. i.e. JUnit for TDD, JBehave for BDD
     
    4,55%
    1
    They seem to be a better "fit for purpose"
     
    4,55%
    1
    I'm using TestNG when working on my personal projects, and JUnit at work because it's the tool my team chose
     
    4,55%
    1
    History
     
    4,55%
    1
    Because of the MockitoJUnitRunner used to test/inject Spring Contexts.
     
    4,55%
    1
    Junit because i use DBunit too. Scala Test because we use it in our team
     
    4,55%
    1
    Each tool has it's domain where it plays better than others.
     
    4,55%
    1
    My team is using JUnit for unit testing in most of our projects
     
    4,55%
    1
    other teams/projects use it
     
    4,55%
    1
    - Spock - it better matches testing Groovy specific things - Spock - nice things can be done in Groovy - JBehave/Cucumber - more readable in acceptance tests - JBehave/Cucumber - easier to use by non-programmers - JUnit - some just maintained old projects
     
    4,55%
    1
    Spock mocking and its power
     
    4,55%
    1
    pokaż więcej odpowiedzi
    Wypełnienia: 62
  • 11. What assertions do you use?

    Pokaż odpowiedzi Ukryj odpowiedzi
    Odpowiedź   % Liczba odp.
    Original TestNG assertions
     
    62,90%
    39
    Hamcrest
     
    20,97%
    13
    FEST Fluent Assertions
     
    33,87%
    21
    JUnit assertions
     
    17,74%
    11
    TestNG new assertions
     
    24,19%
    15
    Other
     
    1,61%
    Wypełnienia: 62
  • 12. What mocking framework do you use when writing TestNG tests?

    Pokaż odpowiedzi Ukryj odpowiedzi
    Odpowiedź   % Liczba odp.
    EasyMock
     
    20,69%
    12
    Mockito
     
    74,14%
    43
    JMock
     
    5,17%
    3
    PowerMock
     
    12,07%
    7
    JMockit
     
    6,90%
    4
    Groovy built-in mocks
     
    5,17%
    3
    ScalaMock
     
    1,72%
    1
    I do not use any mocking framework
     
    18,97%
    11
    Other
     
    3,45%
    Wypełnienia: 58
  • 13. Which code coverage tool do you use?

    Pokaż odpowiedzi Ukryj odpowiedzi
    Odpowiedź   % Liczba odp.
    Clover
     
    8,33%
    5
    Emma
     
    28,33%
    17
    Cobertura
     
    45,00%
    27
    JaCoCo
     
    20,00%
    12
    Built-In IDE (Eclipse, IntelliJ) coverage tool
     
    23,33%
    14
    I do not use any coverage tool
     
    28,33%
    17
    Other
     
    5,00%
    Wypełnienia: 60
  • 3 strona

  • Only 4 more questions and you can go back to work ...or whatever you were doing ;)
  • 14. How do you learn using TestNG?

    By reading:
    Pokaż odpowiedzi Ukryj odpowiedzi
    Odpowiedź   % Liczba odp.
    documentation on http://testng.org
     
    87,10%
    54
    other web resources - e.g. blog, articles
     
    48,39%
    30
    books
     
    30,65%
    19
    TestNG mailing list (or asking questions there)
     
    27,42%
    17
    Stack Overflow (or asking questions there)
     
    35,48%
    22
    Other
     
    4,84%
    Wypełnienia: 62
  • 15. What do you think about TestNG documentation?

    I mean original documentation at http://testng.org
    Pokaż odpowiedzi Ukryj odpowiedzi
    Odpowiedź   % Liczba odp.
    Good
     
    66,67%
    40
    So-So
     
    30,00%
    18
    Bad
     
    3,33%
    2
    Wypełnienia: 60
  • 16. Who are you? *

    Pokaż odpowiedzi Ukryj odpowiedzi
    Odpowiedź   % Liczba odp.
    Developer
     
    88,71%
    55
    Tester
     
    22,58%
    14
    Other
     
    4,84%
    Wypełnienia: 62
  • 17. This place is for you to say something about TestNG or about this survey.

    Pokaż odpowiedzi Ukryj odpowiedzi
    Odpowiedź   % Liczba odp.
    Overall quality of TestNG needs to improve. This starts from the plain code quality over website/documentation to reliability during productional use (i.e. swallowed errors etc.)
     
    7,69%
    1
    I fucking love it!
     
    7,69%
    1
    TestNG is far superior to JUnit. For the Java world, its really been a wonderful asset that doesn't get enough appreciation. Java has reached EOL for me in terms of my primary development focus and where I'm leading my group. Unfortunately that means TestNG will play a smaller role we move away from our legacy stack.
     
    7,69%
    1
    The most positive aspect I see about TestNG now is that it remains actively developed, while JUnit development seems to be dying out.
     
    7,69%
    1
    Love it!
     
    7,69%
    1
    About survey: I think, it will be good question about not only what features we have already using, but also what features we are going to use in nearly future. Now I'm not using new assertions, but I'm going to use it in new project, and later also refactor "old" test too. About TestNg: I think this is the best testing framework for "JVM World", with great, supporting community (and ofcourse Cedric activity in this community is great), but still JUnit has better "PR" (or marketing) in developer's world.
     
    7,69%
    1
    The whole reason I use TestNG is it's flexibility in not forcing isolated unit tests(when a scenario does not warrant the overhead of creating such) but a flexible enough annotation framework to create dependency chains.
     
    7,69%
    1
    15 . What do you think about TestNG documentation? misses: I don't know option.
     
    7,69%
    1
    I forgot to mention a feature...better integration with mocking frameworks such as Mockito allowing easier setup of common frameworks...
     
    7,69%
    1
    I chose Testng as, what I believed would be the most flexible framework for the future, however, I have yet to take advantage of (any) of its features. Hopefully, as my test environment evolves, I will be able to make use of the more powerful features.
     
    7,69%
    1
    Keep moving forward.
     
    7,69%
    1
    TestNG providers many features i really love: especially groups, parameterized tests using data providers. One thing I forgot to mention when writing about the featurues I'd lovde to see in future releases is build-in exception handling - like in http://code.google.com/p/catch-exception/
     
    7,69%
    1
    keep up the good work
     
    7,69%
    1
    Wypełnienia: 62
  • Thank you, that is it!
Komentarze

Przekaż swoje uwagi na temat ankiety zostawiając komentarz


 
 
Formularze online